title: Wolf: Captioning Everything with a World Summarization Framework
summary: We propose Wolf, a WOrLd summarization Framework for accurate video
  captioning. Wolf is an automated captioning framework that adopts a
  mixture-of-experts approach, leveraging complementary strengths of Vision
  Language Models (VLMs). By utilizing both image and video models, our framework
  captures different levels of information and summarizes them efficiently. Our
  approach can be applied to enhance video understanding, auto-labeling, and
  captioning. To evaluate caption quality, we introduce CapScore, an LLM-based
  metric to assess the similarity and quality of generated captions compared to
  the ground truth captions. We further build four human-annotated datasets in
  three domains: autonomous driving, general scenes, and robotics, to facilitate
  comprehensive comparisons. We show that Wolf achieves superior captioning
  performance compared to state-of-the-art approaches from the research community
  (VILA1.5, CogAgent) and commercial solutions (Gemini-Pro-1.5, GPT-4V). For
  instance, in comparison with GPT-4V, Wolf improves CapScore both quality-wise
  by 55.6% and similarity-wise by 77.4% on challenging driving videos. Finally,
  we establish a benchmark for video captioning and introduce a leaderboard,
  aiming to accelerate advancements in video understanding, captioning, and data
  alignment. Leaderboard: https://wolfv0.github.io/leaderboard.html.
paper_authors._id: 66a70891aa9822109f7d4ea3
paper_authors.name: Boyi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-29T08:40:58.724Z
paper_authors._id: 620dd3888528f797e88cb9b5
paper_authors.avatarUrl: /avatars/af04728788d78fe7d6375e19e32a535e.svg
paper_authors.fullname: Boyi Li
paper_authors.type: user
paper_authors.user: Boyiliee
paper_authors._id: 66a70891aa9822109f7d4ea4
paper_authors.name: Ligeng Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-29T08:41:05.160Z
paper_authors._id: 62b4b5beb25cb80fcf278354
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b4b5beb25cb80fcf278354/7uGNOeE91paZikhK2QFgp.jpeg
paper_authors.fullname: Ligeng Zhu
paper_authors.type: user
paper_authors.user: Ligeng-Zhu
paper_authors._id: 66a70891aa9822109f7d4ea5
paper_authors.name: Ran Tian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T09:23:13.956Z
paper_authors._id: 6535786a24c3063696804cdf
paper_authors.avatarUrl: /avatars/2e59a26b691a9ab9825a015c7d47f9aa.svg
paper_authors.fullname: Ran (Thomas) Tian
paper_authors.type: user
paper_authors.user: thomasrantian
paper_authors._id: 66a70891aa9822109f7d4ea6
paper_authors.name: Shuhan Tan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-29T08:41:17.789Z
paper_authors._id: 64b64debeb9a833e08d079fd
paper_authors.avatarUrl: /avatars/62ad6f5a8c1b69252e855ef26cc4e7c2.svg
paper_authors.fullname: Shuhan Tan
paper_authors.type: user
paper_authors.user: tanshh97
paper_authors._id: 66a70891aa9822109f7d4ea7
paper_authors.name: Yuxiao Chen
paper_authors._id: 66a70891aa9822109f7d4ea8
paper_authors.name: Yao Lu
paper_authors._id: 66a70891aa9822109f7d4ea9
paper_authors.name: Yin Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-29T08:42:18.154Z
paper_authors._id: 649f05367b57fab3a5b27c8b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649f05367b57fab3a5b27c8b/UDJB4yqF2NmaRwCyTOfcl.jpeg
paper_authors.fullname: Yin Cui
paper_authors.type: user
paper_authors.user: richardaecn
paper_authors._id: 66a70891aa9822109f7d4eaa
paper_authors.name: Sushant Veer
paper_authors._id: 66a70891aa9822109f7d4eab
paper_authors.name: Max Ehrlich
paper_authors._id: 66a70891aa9822109f7d4eac
paper_authors.name: Jonah Philion
paper_authors._id: 66a70891aa9822109f7d4ead
paper_authors.name: Xinshuo Weng
paper_authors._id: 66a70891aa9822109f7d4eae
paper_authors.name: Fuzhao Xue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-29T08:42:58.983Z
paper_authors._id: 64a52314978242bb77e05790
paper_authors.avatarUrl: /avatars/e97206c653de3446266c80c2943c0a6f.svg
paper_authors.fullname: Fuzhao Xue
paper_authors.type: user
paper_authors.user: fuzhao
paper_authors._id: 66a70891aa9822109f7d4eaf
paper_authors.name: Andrew Tao
paper_authors._id: 66a70891aa9822109f7d4eb0
paper_authors.name: Ming-Yu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-29T08:43:42.822Z
paper_authors._id: 62f049afdf4b93aad5c7f2d6
paper_authors.avatarUrl: /avatars/e272e58ad996733d7098e50248e5b57e.svg
paper_authors.fullname: Ming-Yu Liu
paper_authors.type: user
paper_authors.user: mingyuliutw
paper_authors._id: 66a70891aa9822109f7d4eb1
paper_authors.name: Sanja Fidler
paper_authors._id: 66a70891aa9822109f7d4eb2
paper_authors.name: Boris Ivanovic
paper_authors._id: 66a70891aa9822109f7d4eb3
paper_authors.name: Trevor Darrell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-29T08:43:58.706Z
paper_authors._id: 64cbdf02f103036e23d1c7f3
paper_authors.avatarUrl: /avatars/496069463900dea20929b57381182d39.svg
paper_authors.fullname: Trevor Darrell
paper_authors.type: user
paper_authors.user: trevordarrell
paper_authors._id: 66a70891aa9822109f7d4eb4
paper_authors.name: Jitendra Malik
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-29T08:44:05.344Z
paper_authors._id: 65369a95605a07338de78ab0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sGFjOjLT2akN-sn5beVWL.jpeg
paper_authors.fullname: Jitendra Malik 
paper_authors.type: user
paper_authors.user: jitendra1995
paper_authors._id: 66a70891aa9822109f7d4eb5
paper_authors.name: Song Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-29T08:44:20.741Z
paper_authors._id: 63797f727df2fefdcaf3ff7e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668906853549-noauth.jpeg
paper_authors.fullname: Song
paper_authors.type: user
paper_authors.user: songhan
paper_authors._id: 66a70891aa9822109f7d4eb6
paper_authors.name: Marco Pavone
paper_title: Wolf: Captioning Everything with a World Summarization Framework
paper_summary: We propose Wolf, a WOrLd summarization Framework for accurate video
  captioning. Wolf is an automated captioning framework that adopts a
  mixture-of-experts approach, leveraging complementary strengths of Vision
  Language Models (VLMs). By utilizing both image and video models, our framework
  captures different levels of information and summarizes them efficiently. Our
  approach can be applied to enhance video understanding, auto-labeling, and
  captioning. To evaluate caption quality, we introduce CapScore, an LLM-based
  metric to assess the similarity and quality of generated captions compared to
  the ground truth captions. We further build four human-annotated datasets in
  three domains: autonomous driving, general scenes, and robotics, to facilitate
  comprehensive comparisons. We show that Wolf achieves superior captioning
  performance compared to state-of-the-art approaches from the research community
  (VILA1.5, CogAgent) and commercial solutions (Gemini-Pro-1.5, GPT-4V). For
  instance, in comparison with GPT-4V, Wolf improves CapScore both quality-wise
  by 55.6% and similarity-wise by 77.4% on challenging driving videos. Finally,
  we establish a benchmark for video captioning and introduce a leaderboard,
  aiming to accelerate advancements in video understanding, captioning, and data
  alignment. Leaderboard: https://wolfv0.github.io/leaderboard.html.
paper_ai_summary: Wolf is a framework that combines Vision Language Models for efficient video captioning, outperforming existing approaches and introducing a quality assessment metric called CapScore.
paper_ai_keywords: mixture-of-experts Vision Language Models image models video models CapScore video captioning VILA1.5 CogAgent Gemini-Pro-1.5 GPT-4V