Wolf: Captioning Everything with a World Summarization Framework
We propose Wolf, a WOrLd summarization Framework for accurate video
captioning. Wolf is an automated captioning framework that adopts a
mixture-of-experts approach, leveraging complementary strengths of Vision
Language Models (VLMs). By utilizing both image and video models, our framework
captures different levels of information and summarizes them efficiently. Our
approach can be applied to enhance video understanding, auto-labeling, and
captioning. To evaluate caption quality, we introduce CapScore, an LLM-based
metric to assess the similarity and quality of generated captions compared to
the ground truth captions. We further build four human-annotated datasets in
three domains: autonomous driving, general scenes, and robotics, to facilitate
comprehensive comparisons. We show that Wolf achieves superior captioning
performance compared to state-of-the-art approaches from the research community
(VILA1.5, CogAgent) and commercial solutions (Gemini-Pro-1.5, GPT-4V). For
instance, in comparison with GPT-4V, Wolf improves CapScore both quality-wise
by 55.6% and similarity-wise by 77.4% on challenging driving videos. Finally,
we establish a benchmark for video captioning and introduce a leaderboard,
aiming to accelerate advancements in video understanding, captioning, and data
alignment. Leaderboard: https://wolfv0.github.io/leaderboard.html.
66a70891aa9822109f7d4ea3
Boyi Li
admin_assigned
2024-07-29T08:40:58.724Z
620dd3888528f797e88cb9b5
/avatars/af04728788d78fe7d6375e19e32a535e.svg
Boyi Li
user
Boyiliee
66a70891aa9822109f7d4ea4
Ligeng Zhu
admin_assigned
2024-07-29T08:41:05.160Z
62b4b5beb25cb80fcf278354
https://cdn-avatars.huggingface.co/v1/production/uploads/62b4b5beb25cb80fcf278354/7uGNOeE91paZikhK2QFgp.jpeg
Ligeng Zhu
user
Ligeng-Zhu
66a70891aa9822109f7d4ea5
Ran Tian
claimed_verified
2024-12-11T09:23:13.956Z
6535786a24c3063696804cdf
/avatars/2e59a26b691a9ab9825a015c7d47f9aa.svg
Ran (Thomas) Tian
user
thomasrantian
66a70891aa9822109f7d4ea6
Shuhan Tan
admin_assigned
2024-07-29T08:41:17.789Z
64b64debeb9a833e08d079fd
/avatars/62ad6f5a8c1b69252e855ef26cc4e7c2.svg
Shuhan Tan
user
tanshh97
66a70891aa9822109f7d4ea7
Yuxiao Chen
66a70891aa9822109f7d4ea8
Yao Lu
66a70891aa9822109f7d4ea9
Yin Cui
admin_assigned
2024-07-29T08:42:18.154Z
649f05367b57fab3a5b27c8b
https://cdn-avatars.huggingface.co/v1/production/uploads/649f05367b57fab3a5b27c8b/UDJB4yqF2NmaRwCyTOfcl.jpeg
Yin Cui
user
richardaecn
66a70891aa9822109f7d4eaa
Sushant Veer
66a70891aa9822109f7d4eab
Max Ehrlich
66a70891aa9822109f7d4eac
Jonah Philion
66a70891aa9822109f7d4ead
Xinshuo Weng
66a70891aa9822109f7d4eae
Fuzhao Xue
admin_assigned
2024-07-29T08:42:58.983Z
64a52314978242bb77e05790
/avatars/e97206c653de3446266c80c2943c0a6f.svg
Fuzhao Xue
user
fuzhao
66a70891aa9822109f7d4eaf
Andrew Tao
66a70891aa9822109f7d4eb0
Ming-Yu Liu
admin_assigned
2024-07-29T08:43:42.822Z
62f049afdf4b93aad5c7f2d6
/avatars/e272e58ad996733d7098e50248e5b57e.svg
Ming-Yu Liu
user
mingyuliutw
66a70891aa9822109f7d4eb1
Sanja Fidler
66a70891aa9822109f7d4eb2
Boris Ivanovic
66a70891aa9822109f7d4eb3
Trevor Darrell
admin_assigned
2024-07-29T08:43:58.706Z
64cbdf02f103036e23d1c7f3
/avatars/496069463900dea20929b57381182d39.svg
Trevor Darrell
user
trevordarrell
66a70891aa9822109f7d4eb4
Jitendra Malik
admin_assigned
2024-07-29T08:44:05.344Z
65369a95605a07338de78ab0
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sGFjOjLT2akN-sn5beVWL.jpeg
Jitendra Malik 
user
jitendra1995
66a70891aa9822109f7d4eb5
Song Han
admin_assigned
2024-07-29T08:44:20.741Z
63797f727df2fefdcaf3ff7e
https://cdn-avatars.huggingface.co/v1/production/uploads/1668906853549-noauth.jpeg
Song
user
songhan
66a70891aa9822109f7d4eb6
Marco Pavone
Wolf: Captioning Everything with a World Summarization Framework
We propose Wolf, a WOrLd summarization Framework for accurate video
captioning. Wolf is an automated captioning framework that adopts a
mixture-of-experts approach, leveraging complementary strengths of Vision
Language Models (VLMs). By utilizing both image and video models, our framework
captures different levels of information and summarizes them efficiently. Our
approach can be applied to enhance video understanding, auto-labeling, and
captioning. To evaluate caption quality, we introduce CapScore, an LLM-based
metric to assess the similarity and quality of generated captions compared to
the ground truth captions. We further build four human-annotated datasets in
three domains: autonomous driving, general scenes, and robotics, to facilitate
comprehensive comparisons. We show that Wolf achieves superior captioning
performance compared to state-of-the-art approaches from the research community
(VILA1.5, CogAgent) and commercial solutions (Gemini-Pro-1.5, GPT-4V). For
instance, in comparison with GPT-4V, Wolf improves CapScore both quality-wise
by 55.6% and similarity-wise by 77.4% on challenging driving videos. Finally,
we establish a benchmark for video captioning and introduce a leaderboard,
aiming to accelerate advancements in video understanding, captioning, and data
alignment. Leaderboard: https://wolfv0.github.io/leaderboard.html.
Wolf is a framework that combines Vision Language Models for efficient video captioning, outperforming existing approaches and introducing a quality assessment metric called CapScore.
mixture-of-experts Vision Language Models image models video models CapScore video captioning VILA1.5 CogAgent Gemini-Pro-1.5 GPT-4V