Streaming Dense Video Captioning
An ideal model for dense video captioning -- predicting captions localized
temporally in a video -- should be able to handle long input videos, predict
rich, detailed textual descriptions, and be able to produce outputs before
processing the entire video. Current state-of-the-art models, however, process
a fixed number of downsampled frames, and make a single full prediction after
seeing the whole video. We propose a streaming dense video captioning model
that consists of two novel components: First, we propose a new memory module,
based on clustering incoming tokens, which can handle arbitrarily long videos
as the memory is of a fixed size. Second, we develop a streaming decoding
algorithm that enables our model to make predictions before the entire video
has been processed. Our model achieves this streaming ability, and
significantly improves the state-of-the-art on three dense video captioning
benchmarks: ActivityNet, YouCook2 and ViTT. Our code is released at
https://github.com/google-research/scenic.
660c1d625c4bb2bcc484ee9c
Xingyi Zhou
660c1d625c4bb2bcc484ee9d
Anurag Arnab
660c1d625c4bb2bcc484ee9e
Shyamal Buch
660c1d625c4bb2bcc484ee9f
Shen Yan
660c1d625c4bb2bcc484eea0
Austin Myers
660c1d625c4bb2bcc484eea1
Xuehan Xiong
660c1d625c4bb2bcc484eea2
Arsha Nagrani
660c1d625c4bb2bcc484eea3
Cordelia Schmid
Streaming Dense Video Captioning
An ideal model for dense video captioning -- predicting captions localized
temporally in a video -- should be able to handle long input videos, predict
rich, detailed textual descriptions, and be able to produce outputs before
processing the entire video. Current state-of-the-art models, however, process
a fixed number of downsampled frames, and make a single full prediction after
seeing the whole video. We propose a streaming dense video captioning model
that consists of two novel components: First, we propose a new memory module,
based on clustering incoming tokens, which can handle arbitrarily long videos
as the memory is of a fixed size. Second, we develop a streaming decoding
algorithm that enables our model to make predictions before the entire video
has been processed. Our model achieves this streaming ability, and
significantly improves the state-of-the-art on three dense video captioning
benchmarks: ActivityNet, YouCook2 and ViTT. Our code is released at
https://github.com/google-research/scenic.
A streaming dense video captioning model with a novel memory module and decoding algorithm improves state-of-the-art performance on multiple benchmarks while handling long videos efficiently.
memory module streaming decoding algorithm dense video captioning ActivityNet YouCook2 ViTT