title: AuroraCap: Efficient, Performant Video Detailed Captioning and a New
    Benchmark
summary: Video detailed captioning is a key task which aims to generate comprehensive
  and coherent textual descriptions of video content, benefiting both video
  understanding and generation. In this paper, we propose AuroraCap, a video
  captioner based on a large multimodal model. We follow the simplest
  architecture design without additional parameters for temporal modeling. To
  address the overhead caused by lengthy video sequences, we implement the token
  merging strategy, reducing the number of input visual tokens. Surprisingly, we
  found that this strategy results in little performance loss. AuroraCap shows
  superior performance on various video and image captioning benchmarks, for
  example, obtaining a CIDEr of 88.9 on Flickr30k, beating GPT-4V (55.3) and
  Gemini-1.5 Pro (82.2). However, existing video caption benchmarks only include
  simple descriptions, consisting of a few dozen words, which limits research in
  this field. Therefore, we develop VDC, a video detailed captioning benchmark
  with over one thousand carefully annotated structured captions. In addition, we
  propose a new LLM-assisted metric VDCscore for bettering evaluation, which
  adopts a divide-and-conquer strategy to transform long caption evaluation into
  multiple short question-answer pairs. With the help of human Elo ranking, our
  experiments show that this benchmark better correlates with human judgments of
  video detailed captioning quality.
paper_authors._id: 67033b94ac13ca8bf9489f29
paper_authors.name: Wenhao Chai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T07:19:31.187Z
paper_authors._id: 637c7503fe115289cfecbe6b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676361945047-637c7503fe115289cfecbe6b.jpeg
paper_authors.fullname: Wenhao Chai
paper_authors.type: user
paper_authors.user: wchai
paper_authors._id: 67033b94ac13ca8bf9489f2a
paper_authors.name: Enxin Song
paper_authors._id: 67033b94ac13ca8bf9489f2b
paper_authors.name: Yilun Du
paper_authors._id: 67033b94ac13ca8bf9489f2c
paper_authors.name: Chenlin Meng
paper_authors._id: 67033b94ac13ca8bf9489f2d
paper_authors.name: Vashisht Madhavan
paper_authors._id: 67033b94ac13ca8bf9489f2e
paper_authors.name: Omer Bar-Tal
paper_authors._id: 67033b94ac13ca8bf9489f2f
paper_authors.name: Jeng-Neng Hwang
paper_authors._id: 67033b94ac13ca8bf9489f30
paper_authors.name: Saining Xie
paper_authors._id: 67033b94ac13ca8bf9489f31
paper_authors.name: Christopher D. Manning
paper_title: AuroraCap: Efficient, Performant Video Detailed Captioning and a New
    Benchmark
paper_summary: Video detailed captioning is a key task which aims to generate comprehensive
  and coherent textual descriptions of video content, benefiting both video
  understanding and generation. In this paper, we propose AuroraCap, a video
  captioner based on a large multimodal model. We follow the simplest
  architecture design without additional parameters for temporal modeling. To
  address the overhead caused by lengthy video sequences, we implement the token
  merging strategy, reducing the number of input visual tokens. Surprisingly, we
  found that this strategy results in little performance loss. AuroraCap shows
  superior performance on various video and image captioning benchmarks, for
  example, obtaining a CIDEr of 88.9 on Flickr30k, beating GPT-4V (55.3) and
  Gemini-1.5 Pro (82.2). However, existing video caption benchmarks only include
  simple descriptions, consisting of a few dozen words, which limits research in
  this field. Therefore, we develop VDC, a video detailed captioning benchmark
  with over one thousand carefully annotated structured captions. In addition, we
  propose a new LLM-assisted metric VDCscore for bettering evaluation, which
  adopts a divide-and-conquer strategy to transform long caption evaluation into
  multiple short question-answer pairs. With the help of human Elo ranking, our
  experiments show that this benchmark better correlates with human judgments of
  video detailed captioning quality.
paper_ai_summary: AuroraCap, a large multimodal video captioner using token merging, achieves superior performance on video and image captioning benchmarks, and introduces a new benchmark and evaluation metric for video detailed captioning.
paper_ai_keywords: multimodal model token merging CIDEr GPT-4V Gemini-1.5 Pro video detailed captioning benchmark VDC LLM-assisted metric VDCscore divide-and-conquer strategy Elo ranking