Chat with AI: The Surprising Turn of Real-time Video Communication from
  Human to AI
AI Video Chat emerges as a new paradigm for Real-time Communication (RTC),
where one peer is not a human, but a Multimodal Large Language Model (MLLM).
This makes interaction between humans and AI more intuitive, as if chatting
face-to-face with a real person. However, this poses significant challenges to
latency, because the MLLM inference takes up most of the response time, leaving
very little time for video streaming. Due to network uncertainty and
instability, transmission latency becomes a critical bottleneck preventing AI
from being like a real person. To address this, we propose Artic, an
AI-oriented Real-time Communication framework, exploring the network
requirement shift from "humans watching video" to "AI understanding video". To
reduce bitrate dramatically while maintaining MLLM accuracy, we propose
Context-Aware Video Streaming that recognizes the importance of each video
region for chat and allocates bitrate almost exclusively to chat-important
regions. To avoid packet retransmission, we propose Loss-Resilient Adaptive
Frame Rate that leverages previous frames to substitute for lost/delayed frames
while avoiding bitrate waste. To evaluate the impact of video streaming quality
on MLLM accuracy, we build the first benchmark, named Degraded Video
Understanding Benchmark (DeViBench). Finally, we discuss some open questions
and ongoing solutions for AI Video Chat.
6886fab77d7a19a208cdf212
Jiangkai Wu
claimed_verified
2025-07-28T08:44:35.831Z
67e43f9b29d49781111e4013
https://cdn-avatars.huggingface.co/v1/production/uploads/67e43f9b29d49781111e4013/UXMNGH3-PsVEE3oazmGx2.jpeg
Jiangkai
user
keyonN
6886fab77d7a19a208cdf213
Zhiyuan Ren
6886fab77d7a19a208cdf214
Liming Liu
6886fab77d7a19a208cdf215
Xinggong Zhang
Chat with AI: The Surprising Turn of Real-time Video Communication from
  Human to AI
AI Video Chat emerges as a new paradigm for Real-time Communication (RTC),
where one peer is not a human, but a Multimodal Large Language Model (MLLM).
This makes interaction between humans and AI more intuitive, as if chatting
face-to-face with a real person. However, this poses significant challenges to
latency, because the MLLM inference takes up most of the response time, leaving
very little time for video streaming. Due to network uncertainty and
instability, transmission latency becomes a critical bottleneck preventing AI
from being like a real person. To address this, we propose Artic, an
AI-oriented Real-time Communication framework, exploring the network
requirement shift from "humans watching video" to "AI understanding video". To
reduce bitrate dramatically while maintaining MLLM accuracy, we propose
Context-Aware Video Streaming that recognizes the importance of each video
region for chat and allocates bitrate almost exclusively to chat-important
regions. To avoid packet retransmission, we propose Loss-Resilient Adaptive
Frame Rate that leverages previous frames to substitute for lost/delayed frames
while avoiding bitrate waste. To evaluate the impact of video streaming quality
on MLLM accuracy, we build the first benchmark, named Degraded Video
Understanding Benchmark (DeViBench). Finally, we discuss some open questions
and ongoing solutions for AI Video Chat.
Artic is an AI-oriented real-time communication framework that reduces video streaming latency and bitrate while maintaining MLLM accuracy through context-aware streaming and loss-resilient adaptive frame rate.
Multimodal Large Language Model MLLM Context-Aware Video Streaming Loss-Resilient Adaptive Frame Rate Degraded Video Understanding Benchmark DeViBench