title: StreamChat: Chatting with Streaming Video
summary: This paper presents StreamChat, a novel approach that enhances the
  interaction capabilities of Large Multimodal Models (LMMs) with streaming video
  content. In streaming interaction scenarios, existing methods rely solely on
  visual information available at the moment a question is posed, resulting in
  significant delays as the model remains unaware of subsequent changes in the
  streaming video. StreamChat addresses this limitation by innovatively updating
  the visual context at each decoding step, ensuring that the model utilizes
  up-to-date video content throughout the decoding process. Additionally, we
  introduce a flexible and efficient crossattention-based architecture to process
  dynamic streaming inputs while maintaining inference efficiency for streaming
  interactions. Furthermore, we construct a new dense instruction dataset to
  facilitate the training of streaming interaction models, complemented by a
  parallel 3D-RoPE mechanism that encodes the relative temporal information of
  visual and text tokens. Experimental results demonstrate that StreamChat
  achieves competitive performance on established image and video benchmarks and
  exhibits superior capabilities in streaming interaction scenarios compared to
  state-of-the-art video LMM.
paper_authors._id: 675a9f2184252325bb60445c
paper_authors.name: Jihao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-12T09:00:15.170Z
paper_authors._id: 630458fad14428368d1d0676
paper_authors.avatarUrl: /avatars/05a0b0b75a6a9bdbc4185cc0880abf5e.svg
paper_authors.fullname: LIU Jihao
paper_authors.type: user
paper_authors.user: jjjjh
paper_authors._id: 675a9f2184252325bb60445d
paper_authors.name: Zhiding Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-12T09:07:31.665Z
paper_authors._id: 66c8037c737ba92ae3fe0322
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66c8037c737ba92ae3fe0322/WR_Yh5DWOVVh7IFlF24NM.jpeg
paper_authors.fullname: Zhiding Yu
paper_authors.type: user
paper_authors.user: Zhiding
paper_authors._id: 675a9f2184252325bb60445e
paper_authors.name: Shiyi Lan
paper_authors._id: 675a9f2184252325bb60445f
paper_authors.name: Shihao Wang
paper_authors._id: 675a9f2184252325bb604460
paper_authors.name: Rongyao Fang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-12T09:06:48.987Z
paper_authors._id: 65b8724123d948d884b379b1
paper_authors.avatarUrl: /avatars/ce189d1d8d688c17912f9b869035b2d0.svg
paper_authors.fullname: Rongyao Fang
paper_authors.type: user
paper_authors.user: LucasFang
paper_authors._id: 675a9f2184252325bb604461
paper_authors.name: Jan Kautz
paper_authors._id: 675a9f2184252325bb604462
paper_authors.name: Hongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-12T09:06:29.840Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_authors._id: 675a9f2184252325bb604463
paper_authors.name: Jose M. Alvare
paper_title: StreamChat: Chatting with Streaming Video
paper_summary: This paper presents StreamChat, a novel approach that enhances the
  interaction capabilities of Large Multimodal Models (LMMs) with streaming video
  content. In streaming interaction scenarios, existing methods rely solely on
  visual information available at the moment a question is posed, resulting in
  significant delays as the model remains unaware of subsequent changes in the
  streaming video. StreamChat addresses this limitation by innovatively updating
  the visual context at each decoding step, ensuring that the model utilizes
  up-to-date video content throughout the decoding process. Additionally, we
  introduce a flexible and efficient crossattention-based architecture to process
  dynamic streaming inputs while maintaining inference efficiency for streaming
  interactions. Furthermore, we construct a new dense instruction dataset to
  facilitate the training of streaming interaction models, complemented by a
  parallel 3D-RoPE mechanism that encodes the relative temporal information of
  visual and text tokens. Experimental results demonstrate that StreamChat
  achieves competitive performance on established image and video benchmarks and
  exhibits superior capabilities in streaming interaction scenarios compared to
  state-of-the-art video LMM.
paper_ai_keywords: StreamChat Large Multimodal Models (LMMs) streaming video visual context decoding step crossattention-based architecture density instruction dataset 3D-RoPE relative temporal information visual and text tokens streaming interaction scenarios state-of-the-art video LMM