title: Any2Caption:Interpreting Any Condition to Caption for Controllable Video
    Generation
summary: To address the bottleneck of accurate user intent interpretation within the
  current video generation community, we present Any2Caption, a novel framework
  for controllable video generation under any condition. The key idea is to
  decouple various condition interpretation steps from the video synthesis step.
  By leveraging modern multimodal large language models (MLLMs), Any2Caption
  interprets diverse inputs--text, images, videos, and specialized cues such as
  region, motion, and camera poses--into dense, structured captions that offer
  backbone video generators with better guidance. We also introduce Any2CapIns, a
  large-scale dataset with 337K instances and 407K conditions for
  any-condition-to-caption instruction tuning. Comprehensive evaluations
  demonstrate significant improvements of our system in controllability and video
  quality across various aspects of existing video generation models. Project
  Page: https://sqwu.top/Any2Cap/
paper_authors._id: 67ec0a7262144ec35d0e571d
paper_authors.name: Shengqiong Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-02T08:23:35.392Z
paper_authors._id: 64c139d867eff857ea51caa8
paper_authors.avatarUrl: /avatars/4b7b3f41c2e2cfa21dd43bbac6e081ae.svg
paper_authors.fullname: Shengqiong Wu
paper_authors.type: user
paper_authors.user: ChocoWu
paper_authors._id: 67ec0a7262144ec35d0e571e
paper_authors.name: Weicai Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T09:36:51.426Z
paper_authors._id: 6360d9f0472131c3bc4f61df
paper_authors.avatarUrl: /avatars/c5d884e5ef19b781e3405aba6dd68ca8.svg
paper_authors.fullname: WeicaiYe
paper_authors.type: user
paper_authors.user: WeicaiYe
paper_authors._id: 67ec0a7262144ec35d0e571f
paper_authors.name: Jiahao Wang
paper_authors._id: 67ec0a7262144ec35d0e5720
paper_authors.name: Quande Liu
paper_authors._id: 67ec0a7262144ec35d0e5721
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T09:38:20.890Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 67ec0a7262144ec35d0e5722
paper_authors.name: Pengfei Wan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T08:29:00.610Z
paper_authors._id: 662f93942510ef5735d7ad00
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png
paper_authors.fullname: magicwpf
paper_authors.type: user
paper_authors.user: magicwpf
paper_authors._id: 67ec0a7262144ec35d0e5723
paper_authors.name: Di Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T09:38:50.116Z
paper_authors._id: 644c8324f02250233d0d67d9
paper_authors.avatarUrl: /avatars/feb39d281457c1750f3eada3c060a23e.svg
paper_authors.fullname: Di Zhang
paper_authors.type: user
paper_authors.user: dizhang
paper_authors._id: 67ec0a7262144ec35d0e5724
paper_authors.name: Kun Gai
paper_authors._id: 67ec0a7262144ec35d0e5725
paper_authors.name: Shuicheng Yan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T09:39:18.884Z
paper_authors._id: 67eaa070b9fa8908e151fd7d
paper_authors.avatarUrl: /avatars/1fe2fd678d2e71099a83a9bcb9ab517e.svg
paper_authors.fullname: shuicheng yan
paper_authors.type: user
paper_authors.user: shuicheng
paper_authors._id: 67ec0a7262144ec35d0e5726
paper_authors.name: Hao Fei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-02T08:23:38.087Z
paper_authors._id: 647773a1168cb428e00e9a8f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647773a1168cb428e00e9a8f/NiRR3ScY6Plzjibfwy1hC.jpeg
paper_authors.fullname: Hao Fei
paper_authors.type: user
paper_authors.user: scofield7419
paper_authors._id: 67ec0a7262144ec35d0e5727
paper_authors.name: Tat-Seng Chua
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T09:39:25.537Z
paper_authors._id: 6570ae84c4993b8fb96f41a8
paper_authors.avatarUrl: /avatars/21f7d79d46ac4df0ecff8eca7678b33f.svg
paper_authors.fullname: Tat-Seng Chua
paper_authors.type: user
paper_authors.user: chuats
paper_title: Any2Caption:Interpreting Any Condition to Caption for Controllable Video
    Generation
paper_summary: To address the bottleneck of accurate user intent interpretation within the
  current video generation community, we present Any2Caption, a novel framework
  for controllable video generation under any condition. The key idea is to
  decouple various condition interpretation steps from the video synthesis step.
  By leveraging modern multimodal large language models (MLLMs), Any2Caption
  interprets diverse inputs--text, images, videos, and specialized cues such as
  region, motion, and camera poses--into dense, structured captions that offer
  backbone video generators with better guidance. We also introduce Any2CapIns, a
  large-scale dataset with 337K instances and 407K conditions for
  any-condition-to-caption instruction tuning. Comprehensive evaluations
  demonstrate significant improvements of our system in controllability and video
  quality across various aspects of existing video generation models. Project
  Page: https://sqwu.top/Any2Cap/
paper_ai_summary: Any2Caption framework improves video generation quality and controllability by decoupling condition interpretation and leveraging multimodal large language models.
paper_ai_keywords: multimodal large language models MLLMs captioning task decoupling video synthesis dense captions structured captions video generators instruction tuning Any2CapIns large-scale dataset controllability video quality task evaluation