title: InstanceCap: Improving Text-to-Video Generation via Instance-aware
    Structured Caption
summary: Text-to-video generation has evolved rapidly in recent years, delivering
  remarkable results. Training typically relies on video-caption paired data,
  which plays a crucial role in enhancing generation performance. However,
  current video captions often suffer from insufficient details, hallucinations
  and imprecise motion depiction, affecting the fidelity and consistency of
  generated videos. In this work, we propose a novel instance-aware structured
  caption framework, termed InstanceCap, to achieve instance-level and
  fine-grained video caption for the first time. Based on this scheme, we design
  an auxiliary models cluster to convert original video into instances to enhance
  instance fidelity. Video instances are further used to refine dense prompts
  into structured phrases, achieving concise yet precise descriptions.
  Furthermore, a 22K InstanceVid dataset is curated for training, and an
  enhancement pipeline that tailored to InstanceCap structure is proposed for
  inference. Experimental results demonstrate that our proposed InstanceCap
  significantly outperform previous models, ensuring high fidelity between
  captions and videos while reducing hallucinations.
paper_authors._id: 675ba7108e73e58d6574019e
paper_authors.name: Tiehan Fan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:22:20.639Z
paper_authors._id: 65927f3b754092f6b1e187a7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65927f3b754092f6b1e187a7/gUrNvIQHmsl1vLwSUxpmL.jpeg
paper_authors.fullname: tiehan fan
paper_authors.type: user
paper_authors.user: AnonMegumi
paper_authors._id: 675ba7108e73e58d6574019f
paper_authors.name: Kepan Nan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:05:10.506Z
paper_authors._id: 66271e50f82ffd771a2720ee
paper_authors.avatarUrl: /avatars/34f0f5f954a8c8ff41f86320813b034a.svg
paper_authors.fullname: nankepan
paper_authors.type: user
paper_authors.user: nankepan
paper_authors._id: 675ba7108e73e58d657401a0
paper_authors.name: Rui Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:42:20.696Z
paper_authors._id: 660a7ecf14cfe4973e0acfe1
paper_authors.avatarUrl: /avatars/e488058397f2b7a617515a4f721a9a00.svg
paper_authors.fullname: Rui Xie
paper_authors.type: user
paper_authors.user: SherryX
paper_authors._id: 675ba7108e73e58d657401a1
paper_authors.name: Penghao Zhou
paper_authors._id: 675ba7108e73e58d657401a2
paper_authors.name: Zhenheng Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:05:24.220Z
paper_authors._id: 6421183b69a2c2933882d652
paper_authors.avatarUrl: /avatars/66813a8fa22915087cccd4dbfb945ca7.svg
paper_authors.fullname: Zhenheng Yang
paper_authors.type: user
paper_authors.user: zhenheny
paper_authors._id: 675ba7108e73e58d657401a3
paper_authors.name: Chaoyou Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:06:26.985Z
paper_authors._id: 653b268cd1041ca9188954da
paper_authors.avatarUrl: /avatars/2529a821580f5d87ee46867c5dc5b1e5.svg
paper_authors.fullname: BradyFU
paper_authors.type: user
paper_authors.user: BradyFU
paper_authors._id: 675ba7108e73e58d657401a4
paper_authors.name: Xiang Li
paper_authors._id: 675ba7108e73e58d657401a5
paper_authors.name: Jian Yang
paper_authors._id: 675ba7108e73e58d657401a6
paper_authors.name: Ying Tai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:42:22.452Z
paper_authors._id: 65734004769f3ee9bde1af10
paper_authors.avatarUrl: /avatars/d6310ed861972fd691687d8f47413f33.svg
paper_authors.fullname: Ying Tai
paper_authors.type: user
paper_authors.user: yingtai
paper_title: InstanceCap: Improving Text-to-Video Generation via Instance-aware
    Structured Caption
paper_summary: Text-to-video generation has evolved rapidly in recent years, delivering
  remarkable results. Training typically relies on video-caption paired data,
  which plays a crucial role in enhancing generation performance. However,
  current video captions often suffer from insufficient details, hallucinations
  and imprecise motion depiction, affecting the fidelity and consistency of
  generated videos. In this work, we propose a novel instance-aware structured
  caption framework, termed InstanceCap, to achieve instance-level and
  fine-grained video caption for the first time. Based on this scheme, we design
  an auxiliary models cluster to convert original video into instances to enhance
  instance fidelity. Video instances are further used to refine dense prompts
  into structured phrases, achieving concise yet precise descriptions.
  Furthermore, a 22K InstanceVid dataset is curated for training, and an
  enhancement pipeline that tailored to InstanceCap structure is proposed for
  inference. Experimental results demonstrate that our proposed InstanceCap
  significantly outperform previous models, ensuring high fidelity between
  captions and videos while reducing hallucinations.
paper_ai_keywords: instance-aware structured caption framework instance fidelity auxiliary models cluster video instances dense prompts structured phrases InstanceCap InstanceVid dataset