Any2Caption:Interpreting Any Condition to Caption for Controllable Video
  Generation
To address the bottleneck of accurate user intent interpretation within the
current video generation community, we present Any2Caption, a novel framework
for controllable video generation under any condition. The key idea is to
decouple various condition interpretation steps from the video synthesis step.
By leveraging modern multimodal large language models (MLLMs), Any2Caption
interprets diverse inputs--text, images, videos, and specialized cues such as
region, motion, and camera poses--into dense, structured captions that offer
backbone video generators with better guidance. We also introduce Any2CapIns, a
large-scale dataset with 337K instances and 407K conditions for
any-condition-to-caption instruction tuning. Comprehensive evaluations
demonstrate significant improvements of our system in controllability and video
quality across various aspects of existing video generation models. Project
Page: https://sqwu.top/Any2Cap/
67ec0a7262144ec35d0e571d
Shengqiong Wu
claimed_verified
2025-04-02T08:23:35.392Z
64c139d867eff857ea51caa8
/avatars/4b7b3f41c2e2cfa21dd43bbac6e081ae.svg
Shengqiong Wu
user
ChocoWu
67ec0a7262144ec35d0e571e
Weicai Ye
admin_assigned
2025-04-02T09:36:51.426Z
6360d9f0472131c3bc4f61df
/avatars/c5d884e5ef19b781e3405aba6dd68ca8.svg
WeicaiYe
user
WeicaiYe
67ec0a7262144ec35d0e571f
Jiahao Wang
67ec0a7262144ec35d0e5720
Quande Liu
67ec0a7262144ec35d0e5721
Xintao Wang
admin_assigned
2025-04-02T09:38:20.890Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
67ec0a7262144ec35d0e5722
Pengfei Wan
claimed_verified
2025-04-03T08:29:00.610Z
662f93942510ef5735d7ad00
https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png
magicwpf
user
magicwpf
67ec0a7262144ec35d0e5723
Di Zhang
admin_assigned
2025-04-02T09:38:50.116Z
644c8324f02250233d0d67d9
/avatars/feb39d281457c1750f3eada3c060a23e.svg
Di Zhang
user
dizhang
67ec0a7262144ec35d0e5724
Kun Gai
67ec0a7262144ec35d0e5725
Shuicheng Yan
admin_assigned
2025-04-02T09:39:18.884Z
67eaa070b9fa8908e151fd7d
/avatars/1fe2fd678d2e71099a83a9bcb9ab517e.svg
shuicheng yan
user
shuicheng
67ec0a7262144ec35d0e5726
Hao Fei
claimed_verified
2025-04-02T08:23:38.087Z
647773a1168cb428e00e9a8f
https://cdn-avatars.huggingface.co/v1/production/uploads/647773a1168cb428e00e9a8f/NiRR3ScY6Plzjibfwy1hC.jpeg
Hao Fei
user
scofield7419
67ec0a7262144ec35d0e5727
Tat-Seng Chua
admin_assigned
2025-04-02T09:39:25.537Z
6570ae84c4993b8fb96f41a8
/avatars/21f7d79d46ac4df0ecff8eca7678b33f.svg
Tat-Seng Chua
user
chuats
Any2Caption:Interpreting Any Condition to Caption for Controllable Video
  Generation
To address the bottleneck of accurate user intent interpretation within the
current video generation community, we present Any2Caption, a novel framework
for controllable video generation under any condition. The key idea is to
decouple various condition interpretation steps from the video synthesis step.
By leveraging modern multimodal large language models (MLLMs), Any2Caption
interprets diverse inputs--text, images, videos, and specialized cues such as
region, motion, and camera poses--into dense, structured captions that offer
backbone video generators with better guidance. We also introduce Any2CapIns, a
large-scale dataset with 337K instances and 407K conditions for
any-condition-to-caption instruction tuning. Comprehensive evaluations
demonstrate significant improvements of our system in controllability and video
quality across various aspects of existing video generation models. Project
Page: https://sqwu.top/Any2Cap/
Any2Caption framework improves video generation quality and controllability by decoupling condition interpretation and leveraging multimodal large language models.
multimodal large language models MLLMs captioning task decoupling video synthesis dense captions structured captions video generators instruction tuning Any2CapIns large-scale dataset controllability video quality task evaluation