VideoLLM-online: Online Video Large Language Model for Streaming Video
Recent Large Language Models have been enhanced with vision capabilities,
enabling them to comprehend images, videos, and interleaved vision-language
content. However, the learning methods of these large multimodal models
typically treat videos as predetermined clips, making them less effective and
efficient at handling streaming video inputs. In this paper, we propose a novel
Learning-In-Video-Stream (LIVE) framework, which enables temporally aligned,
long-context, and real-time conversation within a continuous video stream. Our
LIVE framework comprises comprehensive approaches to achieve video streaming
dialogue, encompassing: (1) a training objective designed to perform language
modeling for continuous streaming inputs, (2) a data generation scheme that
converts offline temporal annotations into a streaming dialogue format, and (3)
an optimized inference pipeline to speed up the model responses in real-world
video streams. With our LIVE framework, we built VideoLLM-online model upon
Llama-2/Llama-3 and demonstrate its significant advantages in processing
streaming videos. For instance, on average, our model can support streaming
dialogue in a 5-minute video clip at over 10 FPS on an A100 GPU. Moreover, it
also showcases state-of-the-art performance on public offline video benchmarks,
such as recognition, captioning, and forecasting. The code, model, data, and
demo have been made available at https://showlab.github.io/videollm-online.
66711c58459aa2d3b9754c62
Joya Chen
admin_assigned
2024-06-18T08:18:42.692Z
642435a1a3adbc7142c3b0a6
https://cdn-avatars.huggingface.co/v1/production/uploads/642435a1a3adbc7142c3b0a6/wgLT_w9jNWRU3O0jU0646.jpeg
Joya Chen
user
chenjoya
66711c58459aa2d3b9754c63
Zhaoyang Lv
admin_assigned
2024-06-18T14:14:18.922Z
64a5b3ba84f954be0b716ac8
/avatars/f7a1df6229d3c127c609b1f1d5fa9a90.svg
Zhaoyang Lv
user
zhaoyang-lv-meta
66711c58459aa2d3b9754c64
Shiwei Wu
admin_assigned
2024-06-18T14:14:24.827Z
6655ebac65c1fef9818888b9
/avatars/72af0694a26520e873c078f073384f52.svg
Shiwei Wu
user
ILNA
66711c58459aa2d3b9754c65
Kevin Qinghong Lin
admin_assigned
2024-06-18T14:14:37.854Z
64440be5af034cdfd69ca3a7
https://cdn-avatars.huggingface.co/v1/production/uploads/64440be5af034cdfd69ca3a7/qmx24QiDFT29vleCxL9TX.jpeg
Qinghong (Kevin) Lin
user
KevinQHLin
66711c58459aa2d3b9754c66
Chenan Song
admin_assigned
2024-06-18T14:14:48.557Z
64b7a7c36ab5d14ca7fdf304
/avatars/679c417269ad157b50cf9d94354b44df.svg
Song Chenan
user
Chenaniah
66711c58459aa2d3b9754c67
Difei Gao
admin_assigned
2024-06-18T14:14:54.259Z
661abfba70ec567e154fbb00
https://cdn-avatars.huggingface.co/v1/production/uploads/661abfba70ec567e154fbb00/-0EDBVhWSL8_O1ynSwIzG.jpeg
Difei Gao
user
QuStar
66711c58459aa2d3b9754c68
Jia-Wei Liu
admin_assigned
2024-06-18T14:14:59.998Z
661ab240ae3c617a9e48aa2b
/avatars/98d094cc616fa7fe1ecfd610f88fb569.svg
Jia-Wei Liu
user
Tradwife
66711c58459aa2d3b9754c69
Ziteng Gao
admin_assigned
2024-06-18T14:15:38.723Z
648ac3d53470b17ccc90deaf
/avatars/35b678baa7a91959f40f704c5de2a3e1.svg
Ziteng Gao
user
sebgao
66711c58459aa2d3b9754c6a
Dongxing Mao
admin_assigned
2024-06-18T14:15:44.691Z
6331b4719d77091830d2e0c4
/avatars/5d3bd20e03bbd0222b560fe945839785.svg
Dongxing Mao
user
neversa
66711c58459aa2d3b9754c6b
Mike Zheng Shou
admin_assigned
2024-06-18T14:15:50.242Z
661ab3da2b14565c7acccf5c
/avatars/fa4fc03664803e02aede4d4c3d50b393.svg
Mike Zheng Shou
user
AnalMom
VideoLLM-online: Online Video Large Language Model for Streaming Video
Recent Large Language Models have been enhanced with vision capabilities,
enabling them to comprehend images, videos, and interleaved vision-language
content. However, the learning methods of these large multimodal models
typically treat videos as predetermined clips, making them less effective and
efficient at handling streaming video inputs. In this paper, we propose a novel
Learning-In-Video-Stream (LIVE) framework, which enables temporally aligned,
long-context, and real-time conversation within a continuous video stream. Our
LIVE framework comprises comprehensive approaches to achieve video streaming
dialogue, encompassing: (1) a training objective designed to perform language
modeling for continuous streaming inputs, (2) a data generation scheme that
converts offline temporal annotations into a streaming dialogue format, and (3)
an optimized inference pipeline to speed up the model responses in real-world
video streams. With our LIVE framework, we built VideoLLM-online model upon
Llama-2/Llama-3 and demonstrate its significant advantages in processing
streaming videos. For instance, on average, our model can support streaming
dialogue in a 5-minute video clip at over 10 FPS on an A100 GPU. Moreover, it
also showcases state-of-the-art performance on public offline video benchmarks,
such as recognition, captioning, and forecasting. The code, model, data, and
demo have been made available at https://showlab.github.io/videollm-online.
The LIVE framework enhances Large Language Models for real-time video streaming dialogue through a specialized training objective, data generation scheme, and optimized inference pipeline.
Learning-In-Video-Stream (LIVE) multimodal models continuous streaming inputs video streaming dialogue language modeling Llama-2/Llama-3 streaming dialogue format real-time conversation A100 GPU public offline video benchmarks recognition captioning forecasting