title: VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video
    Generation
summary: Recent image-to-video generation methods have demonstrated success in
  enabling control over one or two visual elements, such as camera trajectory or
  object motion. However, these methods are unable to offer control over multiple
  visual elements due to limitations in data and network efficacy. In this paper,
  we introduce VidCRAFT3, a novel framework for precise image-to-video generation
  that enables control over camera motion, object motion, and lighting direction
  simultaneously. To better decouple control over each visual element, we propose
  the Spatial Triple-Attention Transformer, which integrates lighting direction,
  text, and image in a symmetric way. Since most real-world video datasets lack
  lighting annotations, we construct a high-quality synthetic video dataset, the
  VideoLightingDirection (VLD) dataset. This dataset includes lighting direction
  annotations and objects of diverse appearance, enabling VidCRAFT3 to
  effectively handle strong light transmission and reflection effects.
  Additionally, we propose a three-stage training strategy that eliminates the
  need for training data annotated with multiple visual elements (camera motion,
  object motion, and lighting direction) simultaneously. Extensive experiments on
  benchmark datasets demonstrate the efficacy of VidCRAFT3 in producing
  high-quality video content, surpassing existing state-of-the-art methods in
  terms of control granularity and visual coherence. All code and data will be
  publicly available. Project page: https://sixiaozheng.github.io/VidCRAFT3/.
paper_authors._id: 67ac21acaa680a0f8782d273
paper_authors.name: Sixiao Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-14T08:59:27.188Z
paper_authors._id: 6528fe394513680346a500ed
paper_authors.avatarUrl: /avatars/3021a3be7d900e3928cdc375e2a2bf09.svg
paper_authors.fullname: Sixiao Zheng
paper_authors.type: user
paper_authors.user: sxzheng
paper_authors._id: 67ac21acaa680a0f8782d274
paper_authors.name: Zimian Peng
paper_authors._id: 67ac21acaa680a0f8782d275
paper_authors.name: Yanpeng Zhou
paper_authors._id: 67ac21acaa680a0f8782d276
paper_authors.name: Yi Zhu
paper_authors._id: 67ac21acaa680a0f8782d277
paper_authors.name: Hang Xu
paper_authors._id: 67ac21acaa680a0f8782d278
paper_authors.name: Xiangru Huang
paper_authors._id: 67ac21acaa680a0f8782d279
paper_authors.name: Yanwei Fu
paper_title: VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video
    Generation
paper_summary: Recent image-to-video generation methods have demonstrated success in
  enabling control over one or two visual elements, such as camera trajectory or
  object motion. However, these methods are unable to offer control over multiple
  visual elements due to limitations in data and network efficacy. In this paper,
  we introduce VidCRAFT3, a novel framework for precise image-to-video generation
  that enables control over camera motion, object motion, and lighting direction
  simultaneously. To better decouple control over each visual element, we propose
  the Spatial Triple-Attention Transformer, which integrates lighting direction,
  text, and image in a symmetric way. Since most real-world video datasets lack
  lighting annotations, we construct a high-quality synthetic video dataset, the
  VideoLightingDirection (VLD) dataset. This dataset includes lighting direction
  annotations and objects of diverse appearance, enabling VidCRAFT3 to
  effectively handle strong light transmission and reflection effects.
  Additionally, we propose a three-stage training strategy that eliminates the
  need for training data annotated with multiple visual elements (camera motion,
  object motion, and lighting direction) simultaneously. Extensive experiments on
  benchmark datasets demonstrate the efficacy of VidCRAFT3 in producing
  high-quality video content, surpassing existing state-of-the-art methods in
  terms of control granularity and visual coherence. All code and data will be
  publicly available. Project page: https://sixiaozheng.github.io/VidCRAFT3/.
paper_ai_summary: VidCRAFT3, a novel framework, uses a Spatial Triple-Attention Transformer and a synthetic dataset to enable simultaneous control over camera motion, object motion, and lighting direction, producing high-quality video content.
paper_ai_keywords: Spatial Triple-Attention Transformer VideoLightingDirection (VLD) dataset control granularity visual coherence