Dispider: Enabling Video LLMs with Active Real-Time Interaction via
  Disentangled Perception, Decision, and Reaction
Active Real-time interaction with video LLMs introduces a new paradigm for
human-computer interaction, where the model not only understands user intent
but also responds while continuously processing streaming video on the fly.
Unlike offline video LLMs, which analyze the entire video before answering
questions, active real-time interaction requires three capabilities: 1)
Perception: real-time video monitoring and interaction capturing. 2) Decision:
raising proactive interaction in proper situations, 3) Reaction: continuous
interaction with users. However, inherent conflicts exist among the desired
capabilities. The Decision and Reaction require a contrary Perception scale and
grain, and the autoregressive decoding blocks the real-time Perception and
Decision during the Reaction. To unify the conflicted capabilities within a
harmonious system, we present Dispider, a system that disentangles Perception,
Decision, and Reaction. Dispider features a lightweight proactive streaming
video processing module that tracks the video stream and identifies optimal
moments for interaction. Once the interaction is triggered, an asynchronous
interaction module provides detailed responses, while the processing module
continues to monitor the video in the meantime. Our disentangled and
asynchronous design ensures timely, contextually accurate, and computationally
efficient responses, making Dispider ideal for active real-time interaction for
long-duration video streams. Experiments show that Dispider not only maintains
strong performance in conventional video QA tasks, but also significantly
surpasses previous online models in streaming scenario responses, thereby
validating the effectiveness of our architecture. The code and model are
released at https://github.com/Mark12Ding/Dispider.
677cdc004804091804c654c6
Rui Qian
677cdc004804091804c654c7
Shuangrui Ding
admin_assigned
2025-01-07T09:47:38.104Z
65a7c0335e79abfa2ec30c52
/avatars/2f62f83f9c5c4cc9444571f067cd85b7.svg
Shuangrui Ding
user
Mar2Ding
677cdc004804091804c654c8
Xiaoyi Dong
677cdc004804091804c654c9
Pan Zhang
677cdc004804091804c654ca
Yuhang Zang
claimed_verified
2025-01-07T08:41:19.447Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
Yuhang Zang
user
yuhangzang
677cdc004804091804c654cb
Yuhang Cao
admin_assigned
2025-01-07T09:47:16.734Z
65000bef18830fabea469fdd
/avatars/b320c77dfad039d9f9c54127f610d44f.svg
Cao Yuhang
user
yhcao
677cdc004804091804c654cc
Dahua Lin
admin_assigned
2025-01-07T09:47:00.823Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
677cdc004804091804c654cd
Jiaqi Wang
admin_assigned
2025-01-07T09:46:54.181Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
Dispider: Enabling Video LLMs with Active Real-Time Interaction via
  Disentangled Perception, Decision, and Reaction
Active Real-time interaction with video LLMs introduces a new paradigm for
human-computer interaction, where the model not only understands user intent
but also responds while continuously processing streaming video on the fly.
Unlike offline video LLMs, which analyze the entire video before answering
questions, active real-time interaction requires three capabilities: 1)
Perception: real-time video monitoring and interaction capturing. 2) Decision:
raising proactive interaction in proper situations, 3) Reaction: continuous
interaction with users. However, inherent conflicts exist among the desired
capabilities. The Decision and Reaction require a contrary Perception scale and
grain, and the autoregressive decoding blocks the real-time Perception and
Decision during the Reaction. To unify the conflicted capabilities within a
harmonious system, we present Dispider, a system that disentangles Perception,
Decision, and Reaction. Dispider features a lightweight proactive streaming
video processing module that tracks the video stream and identifies optimal
moments for interaction. Once the interaction is triggered, an asynchronous
interaction module provides detailed responses, while the processing module
continues to monitor the video in the meantime. Our disentangled and
asynchronous design ensures timely, contextually accurate, and computationally
efficient responses, making Dispider ideal for active real-time interaction for
long-duration video streams. Experiments show that Dispider not only maintains
strong performance in conventional video QA tasks, but also significantly
surpasses previous online models in streaming scenario responses, thereby
validating the effectiveness of our architecture. The code and model are
released at https://github.com/Mark12Ding/Dispider.
Dispider, a system for active real-time interaction with video LLMs, disentangles and asynchronously processes video for timely, contextually accurate, and efficient responses in streaming scenarios.
active real-time interaction video LLMs human-computer interaction real-time video monitoring interaction capturing proactive interaction autoregressive decoding Dispider lightweight proactive streaming video processing module asynchronous interaction module disentangled design streaming scenario responses video QA tasks