title: VideoLLM-online: Online Video Large Language Model for Streaming Video
summary: Recent Large Language Models have been enhanced with vision capabilities,
  enabling them to comprehend images, videos, and interleaved vision-language
  content. However, the learning methods of these large multimodal models
  typically treat videos as predetermined clips, making them less effective and
  efficient at handling streaming video inputs. In this paper, we propose a novel
  Learning-In-Video-Stream (LIVE) framework, which enables temporally aligned,
  long-context, and real-time conversation within a continuous video stream. Our
  LIVE framework comprises comprehensive approaches to achieve video streaming
  dialogue, encompassing: (1) a training objective designed to perform language
  modeling for continuous streaming inputs, (2) a data generation scheme that
  converts offline temporal annotations into a streaming dialogue format, and (3)
  an optimized inference pipeline to speed up the model responses in real-world
  video streams. With our LIVE framework, we built VideoLLM-online model upon
  Llama-2/Llama-3 and demonstrate its significant advantages in processing
  streaming videos. For instance, on average, our model can support streaming
  dialogue in a 5-minute video clip at over 10 FPS on an A100 GPU. Moreover, it
  also showcases state-of-the-art performance on public offline video benchmarks,
  such as recognition, captioning, and forecasting. The code, model, data, and
  demo have been made available at https://showlab.github.io/videollm-online.
paper_authors._id: 66711c58459aa2d3b9754c62
paper_authors.name: Joya Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T08:18:42.692Z
paper_authors._id: 642435a1a3adbc7142c3b0a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642435a1a3adbc7142c3b0a6/EBmQ7LnfdTdyuhSUti0-d.png
paper_authors.fullname: Joya Chen
paper_authors.type: user
paper_authors.user: chenjoya
paper_authors._id: 66711c58459aa2d3b9754c63
paper_authors.name: Zhaoyang Lv
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T14:14:18.922Z
paper_authors._id: 64a5b3ba84f954be0b716ac8
paper_authors.avatarUrl: /avatars/f7a1df6229d3c127c609b1f1d5fa9a90.svg
paper_authors.fullname: Zhaoyang Lv
paper_authors.type: user
paper_authors.user: zhaoyang-lv-meta
paper_authors._id: 66711c58459aa2d3b9754c64
paper_authors.name: Shiwei Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T14:14:24.827Z
paper_authors._id: 6655ebac65c1fef9818888b9
paper_authors.avatarUrl: /avatars/72af0694a26520e873c078f073384f52.svg
paper_authors.fullname: Shiwei Wu
paper_authors.type: user
paper_authors.user: ILNA
paper_authors._id: 66711c58459aa2d3b9754c65
paper_authors.name: Kevin Qinghong Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T14:14:37.854Z
paper_authors._id: 64440be5af034cdfd69ca3a7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64440be5af034cdfd69ca3a7/qmx24QiDFT29vleCxL9TX.jpeg
paper_authors.fullname: Qinghong (Kevin) Lin
paper_authors.type: user
paper_authors.user: KevinQHLin
paper_authors._id: 66711c58459aa2d3b9754c66
paper_authors.name: Chenan Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T14:14:48.557Z
paper_authors._id: 64b7a7c36ab5d14ca7fdf304
paper_authors.avatarUrl: /avatars/679c417269ad157b50cf9d94354b44df.svg
paper_authors.fullname: Song Chenan
paper_authors.type: user
paper_authors.user: Chenaniah
paper_authors._id: 66711c58459aa2d3b9754c67
paper_authors.name: Difei Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T14:14:54.259Z
paper_authors._id: 661abfba70ec567e154fbb00
paper_authors.avatarUrl: /avatars/46c004650ba15525ccf02d40c9314085.svg
paper_authors.fullname: Difei Gao
paper_authors.type: user
paper_authors.user: QuStar
paper_authors._id: 66711c58459aa2d3b9754c68
paper_authors.name: Jia-Wei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T14:14:59.998Z
paper_authors._id: 661ab240ae3c617a9e48aa2b
paper_authors.avatarUrl: /avatars/98d094cc616fa7fe1ecfd610f88fb569.svg
paper_authors.fullname: Jia-Wei Liu
paper_authors.type: user
paper_authors.user: Tradwife
paper_authors._id: 66711c58459aa2d3b9754c69
paper_authors.name: Ziteng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T14:15:38.723Z
paper_authors._id: 648ac3d53470b17ccc90deaf
paper_authors.avatarUrl: /avatars/35b678baa7a91959f40f704c5de2a3e1.svg
paper_authors.fullname: Ziteng Gao
paper_authors.type: user
paper_authors.user: sebgao
paper_authors._id: 66711c58459aa2d3b9754c6a
paper_authors.name: Dongxing Mao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T14:15:44.691Z
paper_authors._id: 6331b4719d77091830d2e0c4
paper_authors.avatarUrl: /avatars/5d3bd20e03bbd0222b560fe945839785.svg
paper_authors.fullname: Dongxing Mao
paper_authors.type: user
paper_authors.user: neversa
paper_authors._id: 66711c58459aa2d3b9754c6b
paper_authors.name: Mike Zheng Shou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T14:15:50.242Z
paper_authors._id: 661ab3da2b14565c7acccf5c
paper_authors.avatarUrl: /avatars/fa4fc03664803e02aede4d4c3d50b393.svg
paper_authors.fullname: Mike Zheng Shou
paper_authors.type: user
paper_authors.user: AnalMom
paper_title: VideoLLM-online: Online Video Large Language Model for Streaming Video
paper_summary: Recent Large Language Models have been enhanced with vision capabilities,
  enabling them to comprehend images, videos, and interleaved vision-language
  content. However, the learning methods of these large multimodal models
  typically treat videos as predetermined clips, making them less effective and
  efficient at handling streaming video inputs. In this paper, we propose a novel
  Learning-In-Video-Stream (LIVE) framework, which enables temporally aligned,
  long-context, and real-time conversation within a continuous video stream. Our
  LIVE framework comprises comprehensive approaches to achieve video streaming
  dialogue, encompassing: (1) a training objective designed to perform language
  modeling for continuous streaming inputs, (2) a data generation scheme that
  converts offline temporal annotations into a streaming dialogue format, and (3)
  an optimized inference pipeline to speed up the model responses in real-world
  video streams. With our LIVE framework, we built VideoLLM-online model upon
  Llama-2/Llama-3 and demonstrate its significant advantages in processing
  streaming videos. For instance, on average, our model can support streaming
  dialogue in a 5-minute video clip at over 10 FPS on an A100 GPU. Moreover, it
  also showcases state-of-the-art performance on public offline video benchmarks,
  such as recognition, captioning, and forecasting. The code, model, data, and
  demo have been made available at https://showlab.github.io/videollm-online.
paper_ai_keywords: Learning-In-Video-Stream (LIVE) temporally aligned long-context real-time conversation continuous video stream language modeling streaming dialogue format optimized inference pipeline VideoLLM-online Llama-2/Llama-3 Selfue streaming videos FPS A100 GPU offline video benchmarks recognition captioning forecasting