ShareGPT4Video: Improving Video Understanding and Generation with Better
  Captions
We present the ShareGPT4Video series, aiming to facilitate the video
understanding of large video-language models (LVLMs) and the video generation
of text-to-video models (T2VMs) via dense and precise captions. The series
comprises: 1) ShareGPT4Video, 40K GPT4V annotated dense captions of videos with
various lengths and sources, developed through carefully designed data
filtering and annotating strategy. 2) ShareCaptioner-Video, an efficient and
capable captioning model for arbitrary videos, with 4.8M high-quality aesthetic
videos annotated by it. 3) ShareGPT4Video-8B, a simple yet superb LVLM that
reached SOTA performance on three advancing video benchmarks. To achieve this,
taking aside the non-scalable costly human annotators, we find using GPT4V to
caption video with a naive multi-frame or frame-concatenation input strategy
leads to less detailed and sometimes temporal-confused results. We argue the
challenge of designing a high-quality video captioning strategy lies in three
aspects: 1) Inter-frame precise temporal change understanding. 2) Intra-frame
detailed content description. 3) Frame-number scalability for arbitrary-length
videos. To this end, we meticulously designed a differential video captioning
strategy, which is stable, scalable, and efficient for generating captions for
videos with arbitrary resolution, aspect ratios, and length. Based on it, we
construct ShareGPT4Video, which contains 40K high-quality videos spanning a
wide range of categories, and the resulting captions encompass rich world
knowledge, object attributes, camera movements, and crucially, detailed and
precise temporal descriptions of events. Based on ShareGPT4Video, we further
develop ShareCaptioner-Video, a superior captioner capable of efficiently
generating high-quality captions for arbitrary videos...
666274f1287d6c52ffcbdf45
Lin Chen
claimed_verified
2024-06-07T13:30:22.274Z
64b02ec0e5000ae8a572ced5
https://cdn-avatars.huggingface.co/v1/production/uploads/64b02ec0e5000ae8a572ced5/6ifLntBU2ICQK7SW8WxKU.png
Lin Chen
user
Lin-Chen
666274f1287d6c52ffcbdf46
Xilin Wei
admin_assigned
2024-06-07T12:35:11.329Z
62eb70462f0f5e54df42f778
/avatars/456049dba67638d3cdb330cdf383f272.svg
Xilin Wei
user
Wiselnn
666274f1287d6c52ffcbdf47
Jinsong Li
claimed_verified
2024-06-07T11:27:33.715Z
64e5f0c23e220d8f697d1ab0
https://cdn-avatars.huggingface.co/v1/production/uploads/64e5f0c23e220d8f697d1ab0/qD-Egzxs-ZvJT5GXeqE3d.jpeg
Jinsong Li
user
Jinsong-Li
666274f1287d6c52ffcbdf48
Xiaoyi Dong
666274f1287d6c52ffcbdf49
Pan Zhang
666274f1287d6c52ffcbdf4a
Yuhang Zang
admin_assigned
2024-06-07T12:34:04.323Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
Yuhang Zang
user
yuhangzang
666274f1287d6c52ffcbdf4b
Zehui Chen
admin_assigned
2024-06-07T12:34:10.486Z
64892d31cbda0d1cdb956897
/avatars/3cdafe03a8295124636347d15a099aaf.svg
Zehui Chen
user
lovesnowbest
666274f1287d6c52ffcbdf4c
Haodong Duan
admin_assigned
2024-06-07T12:34:16.200Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
666274f1287d6c52ffcbdf4d
Bin Lin
claimed_verified
2024-06-07T11:27:31.681Z
6367a8175bb06007ea099b8f
https://cdn-avatars.huggingface.co/v1/production/uploads/6367a8175bb06007ea099b8f/IjG7HyWyWRlVt_XwRbxRW.jpeg
linbin
user
LanguageBind
666274f1287d6c52ffcbdf4e
Zhenyu Tang
admin_assigned
2024-06-07T12:34:23.236Z
63c09aa88d1175e3399e906e
/avatars/a1b8a89e83275f049f7bf75af9086d84.svg
Zhenyu Tang
user
royjames
666274f1287d6c52ffcbdf4f
Li Yuan
claimed_verified
2025-04-04T07:12:01.726Z
66135a5e50350afe76beebce
/avatars/370a4b83949355feb050c2cb0425c264.svg
yl2488
user
yl2488
666274f1287d6c52ffcbdf50
Yu Qiao
666274f1287d6c52ffcbdf51
Dahua Lin
admin_assigned
2024-06-07T12:34:57.709Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
666274f1287d6c52ffcbdf52
Feng Zhao
666274f1287d6c52ffcbdf53
Jiaqi Wang
claimed_verified
2024-06-09T20:05:01.606Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
ShareGPT4Video: Improving Video Understanding and Generation with Better
  Captions
We present the ShareGPT4Video series, aiming to facilitate the video
understanding of large video-language models (LVLMs) and the video generation
of text-to-video models (T2VMs) via dense and precise captions. The series
comprises: 1) ShareGPT4Video, 40K GPT4V annotated dense captions of videos with
various lengths and sources, developed through carefully designed data
filtering and annotating strategy. 2) ShareCaptioner-Video, an efficient and
capable captioning model for arbitrary videos, with 4.8M high-quality aesthetic
videos annotated by it. 3) ShareGPT4Video-8B, a simple yet superb LVLM that
reached SOTA performance on three advancing video benchmarks. To achieve this,
taking aside the non-scalable costly human annotators, we find using GPT4V to
caption video with a naive multi-frame or frame-concatenation input strategy
leads to less detailed and sometimes temporal-confused results. We argue the
challenge of designing a high-quality video captioning strategy lies in three
aspects: 1) Inter-frame precise temporal change understanding. 2) Intra-frame
detailed content description. 3) Frame-number scalability for arbitrary-length
videos. To this end, we meticulously designed a differential video captioning
strategy, which is stable, scalable, and efficient for generating captions for
videos with arbitrary resolution, aspect ratios, and length. Based on it, we
construct ShareGPT4Video, which contains 40K high-quality videos spanning a
wide range of categories, and the resulting captions encompass rich world
knowledge, object attributes, camera movements, and crucially, detailed and
precise temporal descriptions of events. Based on ShareGPT4Video, we further
develop ShareCaptioner-Video, a superior captioner capable of efficiently
generating high-quality captions for arbitrary videos...
The ShareGPT4Video series enhances video understanding and generation using dense captions and an efficient captioning strategy by addressing temporal and spatial challenges in video annotation.
GPT4V LVLMs T2VMs dense captions captioning model ShareGPT4Video ShareCaptioner-Video differential video captioning temporal change understanding intra-frame description frame-number scalability