StreamChat: Chatting with Streaming Video
This paper presents StreamChat, a novel approach that enhances the
interaction capabilities of Large Multimodal Models (LMMs) with streaming video
content. In streaming interaction scenarios, existing methods rely solely on
visual information available at the moment a question is posed, resulting in
significant delays as the model remains unaware of subsequent changes in the
streaming video. StreamChat addresses this limitation by innovatively updating
the visual context at each decoding step, ensuring that the model utilizes
up-to-date video content throughout the decoding process. Additionally, we
introduce a flexible and efficient crossattention-based architecture to process
dynamic streaming inputs while maintaining inference efficiency for streaming
interactions. Furthermore, we construct a new dense instruction dataset to
facilitate the training of streaming interaction models, complemented by a
parallel 3D-RoPE mechanism that encodes the relative temporal information of
visual and text tokens. Experimental results demonstrate that StreamChat
achieves competitive performance on established image and video benchmarks and
exhibits superior capabilities in streaming interaction scenarios compared to
state-of-the-art video LMM.
675a9f2184252325bb60445c
Jihao Liu
claimed_verified
2024-12-12T09:00:15.170Z
630458fad14428368d1d0676
/avatars/05a0b0b75a6a9bdbc4185cc0880abf5e.svg
LIU Jihao
user
jjjjh
675a9f2184252325bb60445d
Zhiding Yu
admin_assigned
2024-12-12T09:07:31.665Z
66c8037c737ba92ae3fe0322
https://cdn-avatars.huggingface.co/v1/production/uploads/66c8037c737ba92ae3fe0322/WR_Yh5DWOVVh7IFlF24NM.jpeg
Zhiding Yu
user
Zhiding
675a9f2184252325bb60445e
Shiyi Lan
675a9f2184252325bb60445f
Shihao Wang
675a9f2184252325bb604460
Rongyao Fang
admin_assigned
2024-12-12T09:06:48.987Z
65b8724123d948d884b379b1
/avatars/ce189d1d8d688c17912f9b869035b2d0.svg
Rongyao Fang
user
LucasFang
675a9f2184252325bb604461
Jan Kautz
675a9f2184252325bb604462
Hongsheng Li
admin_assigned
2024-12-12T09:06:29.840Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
675a9f2184252325bb604463
Jose M. Alvare
StreamChat: Chatting with Streaming Video
This paper presents StreamChat, a novel approach that enhances the
interaction capabilities of Large Multimodal Models (LMMs) with streaming video
content. In streaming interaction scenarios, existing methods rely solely on
visual information available at the moment a question is posed, resulting in
significant delays as the model remains unaware of subsequent changes in the
streaming video. StreamChat addresses this limitation by innovatively updating
the visual context at each decoding step, ensuring that the model utilizes
up-to-date video content throughout the decoding process. Additionally, we
introduce a flexible and efficient crossattention-based architecture to process
dynamic streaming inputs while maintaining inference efficiency for streaming
interactions. Furthermore, we construct a new dense instruction dataset to
facilitate the training of streaming interaction models, complemented by a
parallel 3D-RoPE mechanism that encodes the relative temporal information of
visual and text tokens. Experimental results demonstrate that StreamChat
achieves competitive performance on established image and video benchmarks and
exhibits superior capabilities in streaming interaction scenarios compared to
state-of-the-art video LMM.
StreamChat enhances Large Multimodal Models' interaction with streaming video by updating visual context at each decoding step and using a crossattention-based architecture with a 3D-RoPE mechanism for temporal information encoding.
Large Multimodal Models streaming video visual context decoding step crossattention-based architecture 3D-RoPE mechanism