title: Streaming Dense Video Captioning
summary: An ideal model for dense video captioning -- predicting captions localized
  temporally in a video -- should be able to handle long input videos, predict
  rich, detailed textual descriptions, and be able to produce outputs before
  processing the entire video. Current state-of-the-art models, however, process
  a fixed number of downsampled frames, and make a single full prediction after
  seeing the whole video. We propose a streaming dense video captioning model
  that consists of two novel components: First, we propose a new memory module,
  based on clustering incoming tokens, which can handle arbitrarily long videos
  as the memory is of a fixed size. Second, we develop a streaming decoding
  algorithm that enables our model to make predictions before the entire video
  has been processed. Our model achieves this streaming ability, and
  significantly improves the state-of-the-art on three dense video captioning
  benchmarks: ActivityNet, YouCook2 and ViTT. Our code is released at
  https://github.com/google-research/scenic.
paper_authors._id: 660c1d625c4bb2bcc484ee9c
paper_authors.name: Xingyi Zhou
paper_authors._id: 660c1d625c4bb2bcc484ee9d
paper_authors.name: Anurag Arnab
paper_authors._id: 660c1d625c4bb2bcc484ee9e
paper_authors.name: Shyamal Buch
paper_authors._id: 660c1d625c4bb2bcc484ee9f
paper_authors.name: Shen Yan
paper_authors._id: 660c1d625c4bb2bcc484eea0
paper_authors.name: Austin Myers
paper_authors._id: 660c1d625c4bb2bcc484eea1
paper_authors.name: Xuehan Xiong
paper_authors._id: 660c1d625c4bb2bcc484eea2
paper_authors.name: Arsha Nagrani
paper_authors._id: 660c1d625c4bb2bcc484eea3
paper_authors.name: Cordelia Schmid
paper_title: Streaming Dense Video Captioning
paper_summary: An ideal model for dense video captioning -- predicting captions localized
  temporally in a video -- should be able to handle long input videos, predict
  rich, detailed textual descriptions, and be able to produce outputs before
  processing the entire video. Current state-of-the-art models, however, process
  a fixed number of downsampled frames, and make a single full prediction after
  seeing the whole video. We propose a streaming dense video captioning model
  that consists of two novel components: First, we propose a new memory module,
  based on clustering incoming tokens, which can handle arbitrarily long videos
  as the memory is of a fixed size. Second, we develop a streaming decoding
  algorithm that enables our model to make predictions before the entire video
  has been processed. Our model achieves this streaming ability, and
  significantly improves the state-of-the-art on three dense video captioning
  benchmarks: ActivityNet, YouCook2 and ViTT. Our code is released at
  https://github.com/google-research/scenic.
paper_ai_summary: A streaming dense video captioning model with a novel memory module and decoding algorithm improves state-of-the-art performance on multiple benchmarks while handling long videos efficiently.
paper_ai_keywords: memory module streaming decoding algorithm dense video captioning ActivityNet YouCook2 ViTT