title: ShareGPT4Video: Improving Video Understanding and Generation with Better
    Captions
summary: We present the ShareGPT4Video series, aiming to facilitate the video
  understanding of large video-language models (LVLMs) and the video generation
  of text-to-video models (T2VMs) via dense and precise captions. The series
  comprises: 1) ShareGPT4Video, 40K GPT4V annotated dense captions of videos with
  various lengths and sources, developed through carefully designed data
  filtering and annotating strategy. 2) ShareCaptioner-Video, an efficient and
  capable captioning model for arbitrary videos, with 4.8M high-quality aesthetic
  videos annotated by it. 3) ShareGPT4Video-8B, a simple yet superb LVLM that
  reached SOTA performance on three advancing video benchmarks. To achieve this,
  taking aside the non-scalable costly human annotators, we find using GPT4V to
  caption video with a naive multi-frame or frame-concatenation input strategy
  leads to less detailed and sometimes temporal-confused results. We argue the
  challenge of designing a high-quality video captioning strategy lies in three
  aspects: 1) Inter-frame precise temporal change understanding. 2) Intra-frame
  detailed content description. 3) Frame-number scalability for arbitrary-length
  videos. To this end, we meticulously designed a differential video captioning
  strategy, which is stable, scalable, and efficient for generating captions for
  videos with arbitrary resolution, aspect ratios, and length. Based on it, we
  construct ShareGPT4Video, which contains 40K high-quality videos spanning a
  wide range of categories, and the resulting captions encompass rich world
  knowledge, object attributes, camera movements, and crucially, detailed and
  precise temporal descriptions of events. Based on ShareGPT4Video, we further
  develop ShareCaptioner-Video, a superior captioner capable of efficiently
  generating high-quality captions for arbitrary videos...
paper_authors._id: 666274f1287d6c52ffcbdf45
paper_authors.name: Lin Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-07T13:30:22.274Z
paper_authors._id: 64b02ec0e5000ae8a572ced5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b02ec0e5000ae8a572ced5/6ifLntBU2ICQK7SW8WxKU.png
paper_authors.fullname: Lin Chen
paper_authors.type: user
paper_authors.user: Lin-Chen
paper_authors._id: 666274f1287d6c52ffcbdf46
paper_authors.name: Xilin Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-07T12:35:11.329Z
paper_authors._id: 62eb70462f0f5e54df42f778
paper_authors.avatarUrl: /avatars/456049dba67638d3cdb330cdf383f272.svg
paper_authors.fullname: Xilin Wei
paper_authors.type: user
paper_authors.user: Wiselnn
paper_authors._id: 666274f1287d6c52ffcbdf47
paper_authors.name: Jinsong Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-07T11:27:33.715Z
paper_authors._id: 64e5f0c23e220d8f697d1ab0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64e5f0c23e220d8f697d1ab0/qD-Egzxs-ZvJT5GXeqE3d.jpeg
paper_authors.fullname: Jinsong Li
paper_authors.type: user
paper_authors.user: Jinsong-Li
paper_authors._id: 666274f1287d6c52ffcbdf48
paper_authors.name: Xiaoyi Dong
paper_authors._id: 666274f1287d6c52ffcbdf49
paper_authors.name: Pan Zhang
paper_authors._id: 666274f1287d6c52ffcbdf4a
paper_authors.name: Yuhang Zang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-07T12:34:04.323Z
paper_authors._id: 63859cf3b2906edaf83af9f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
paper_authors.fullname: Yuhang Zang
paper_authors.type: user
paper_authors.user: yuhangzang
paper_authors._id: 666274f1287d6c52ffcbdf4b
paper_authors.name: Zehui Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-07T12:34:10.486Z
paper_authors._id: 64892d31cbda0d1cdb956897
paper_authors.avatarUrl: /avatars/3cdafe03a8295124636347d15a099aaf.svg
paper_authors.fullname: Zehui Chen
paper_authors.type: user
paper_authors.user: lovesnowbest
paper_authors._id: 666274f1287d6c52ffcbdf4c
paper_authors.name: Haodong Duan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-07T12:34:16.200Z
paper_authors._id: 63ee1379190ddd6214efd73a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
paper_authors.fullname: HAODONG DUAN
paper_authors.type: user
paper_authors.user: KennyUTC
paper_authors._id: 666274f1287d6c52ffcbdf4d
paper_authors.name: Bin Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-07T11:27:31.681Z
paper_authors._id: 6367a8175bb06007ea099b8f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6367a8175bb06007ea099b8f/IjG7HyWyWRlVt_XwRbxRW.jpeg
paper_authors.fullname: linbin
paper_authors.type: user
paper_authors.user: LanguageBind
paper_authors._id: 666274f1287d6c52ffcbdf4e
paper_authors.name: Zhenyu Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-07T12:34:23.236Z
paper_authors._id: 63c09aa88d1175e3399e906e
paper_authors.avatarUrl: /avatars/a1b8a89e83275f049f7bf75af9086d84.svg
paper_authors.fullname: Zhenyu Tang
paper_authors.type: user
paper_authors.user: royjames
paper_authors._id: 666274f1287d6c52ffcbdf4f
paper_authors.name: Li Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:12:01.726Z
paper_authors._id: 66135a5e50350afe76beebce
paper_authors.avatarUrl: /avatars/370a4b83949355feb050c2cb0425c264.svg
paper_authors.fullname: yl2488
paper_authors.type: user
paper_authors.user: yl2488
paper_authors._id: 666274f1287d6c52ffcbdf50
paper_authors.name: Yu Qiao
paper_authors._id: 666274f1287d6c52ffcbdf51
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-07T12:34:57.709Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 666274f1287d6c52ffcbdf52
paper_authors.name: Feng Zhao
paper_authors._id: 666274f1287d6c52ffcbdf53
paper_authors.name: Jiaqi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-09T20:05:01.606Z
paper_authors._id: 64b4eec4faa3181a5eab9c46
paper_authors.avatarUrl: /avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
paper_authors.fullname: Jiaqi Wang
paper_authors.type: user
paper_authors.user: myownskyW7
paper_title: ShareGPT4Video: Improving Video Understanding and Generation with Better
    Captions
paper_summary: We present the ShareGPT4Video series, aiming to facilitate the video
  understanding of large video-language models (LVLMs) and the video generation
  of text-to-video models (T2VMs) via dense and precise captions. The series
  comprises: 1) ShareGPT4Video, 40K GPT4V annotated dense captions of videos with
  various lengths and sources, developed through carefully designed data
  filtering and annotating strategy. 2) ShareCaptioner-Video, an efficient and
  capable captioning model for arbitrary videos, with 4.8M high-quality aesthetic
  videos annotated by it. 3) ShareGPT4Video-8B, a simple yet superb LVLM that
  reached SOTA performance on three advancing video benchmarks. To achieve this,
  taking aside the non-scalable costly human annotators, we find using GPT4V to
  caption video with a naive multi-frame or frame-concatenation input strategy
  leads to less detailed and sometimes temporal-confused results. We argue the
  challenge of designing a high-quality video captioning strategy lies in three
  aspects: 1) Inter-frame precise temporal change understanding. 2) Intra-frame
  detailed content description. 3) Frame-number scalability for arbitrary-length
  videos. To this end, we meticulously designed a differential video captioning
  strategy, which is stable, scalable, and efficient for generating captions for
  videos with arbitrary resolution, aspect ratios, and length. Based on it, we
  construct ShareGPT4Video, which contains 40K high-quality videos spanning a
  wide range of categories, and the resulting captions encompass rich world
  knowledge, object attributes, camera movements, and crucially, detailed and
  precise temporal descriptions of events. Based on ShareGPT4Video, we further
  develop ShareCaptioner-Video, a superior captioner capable of efficiently
  generating high-quality captions for arbitrary videos...
paper_ai_summary: The ShareGPT4Video series enhances video understanding and generation using dense captions and an efficient captioning strategy by addressing temporal and spatial challenges in video annotation.
paper_ai_keywords: GPT4V LVLMs T2VMs dense captions captioning model ShareGPT4Video ShareCaptioner-Video differential video captioning temporal change understanding intra-frame description frame-number scalability