AuroraCap: Efficient, Performant Video Detailed Captioning and a New
  Benchmark
Video detailed captioning is a key task which aims to generate comprehensive
and coherent textual descriptions of video content, benefiting both video
understanding and generation. In this paper, we propose AuroraCap, a video
captioner based on a large multimodal model. We follow the simplest
architecture design without additional parameters for temporal modeling. To
address the overhead caused by lengthy video sequences, we implement the token
merging strategy, reducing the number of input visual tokens. Surprisingly, we
found that this strategy results in little performance loss. AuroraCap shows
superior performance on various video and image captioning benchmarks, for
example, obtaining a CIDEr of 88.9 on Flickr30k, beating GPT-4V (55.3) and
Gemini-1.5 Pro (82.2). However, existing video caption benchmarks only include
simple descriptions, consisting of a few dozen words, which limits research in
this field. Therefore, we develop VDC, a video detailed captioning benchmark
with over one thousand carefully annotated structured captions. In addition, we
propose a new LLM-assisted metric VDCscore for bettering evaluation, which
adopts a divide-and-conquer strategy to transform long caption evaluation into
multiple short question-answer pairs. With the help of human Elo ranking, our
experiments show that this benchmark better correlates with human judgments of
video detailed captioning quality.
67033b94ac13ca8bf9489f29
Wenhao Chai
claimed_verified
2024-10-08T07:19:31.187Z
637c7503fe115289cfecbe6b
https://cdn-avatars.huggingface.co/v1/production/uploads/1676361945047-637c7503fe115289cfecbe6b.jpeg
Wenhao Chai
user
wchai
67033b94ac13ca8bf9489f2a
Enxin Song
claimed_verified
2025-06-25T08:10:54.631Z
63a7b196e27a6dbd4861e275
https://cdn-avatars.huggingface.co/v1/production/uploads/63a7b196e27a6dbd4861e275/QGn4hdDPNuSRgZVKxzDpq.jpeg
EnxinSong
user
Enxin
67033b94ac13ca8bf9489f2b
Yilun Du
67033b94ac13ca8bf9489f2c
Chenlin Meng
67033b94ac13ca8bf9489f2d
Vashisht Madhavan
67033b94ac13ca8bf9489f2e
Omer Bar-Tal
67033b94ac13ca8bf9489f2f
Jeng-Neng Hwang
67033b94ac13ca8bf9489f30
Saining Xie
67033b94ac13ca8bf9489f31
Christopher D. Manning
AuroraCap: Efficient, Performant Video Detailed Captioning and a New
  Benchmark
Video detailed captioning is a key task which aims to generate comprehensive
and coherent textual descriptions of video content, benefiting both video
understanding and generation. In this paper, we propose AuroraCap, a video
captioner based on a large multimodal model. We follow the simplest
architecture design without additional parameters for temporal modeling. To
address the overhead caused by lengthy video sequences, we implement the token
merging strategy, reducing the number of input visual tokens. Surprisingly, we
found that this strategy results in little performance loss. AuroraCap shows
superior performance on various video and image captioning benchmarks, for
example, obtaining a CIDEr of 88.9 on Flickr30k, beating GPT-4V (55.3) and
Gemini-1.5 Pro (82.2). However, existing video caption benchmarks only include
simple descriptions, consisting of a few dozen words, which limits research in
this field. Therefore, we develop VDC, a video detailed captioning benchmark
with over one thousand carefully annotated structured captions. In addition, we
propose a new LLM-assisted metric VDCscore for bettering evaluation, which
adopts a divide-and-conquer strategy to transform long caption evaluation into
multiple short question-answer pairs. With the help of human Elo ranking, our
experiments show that this benchmark better correlates with human judgments of
video detailed captioning quality.
AuroraCap, a large multimodal video captioner using token merging, achieves superior performance on video and image captioning benchmarks, and introduces a new benchmark and evaluation metric for video detailed captioning.
multimodal model token merging CIDEr GPT-4V Gemini-1.5 Pro video detailed captioning benchmark VDC LLM-assisted metric VDCscore divide-and-conquer strategy Elo ranking