title: VFX Creator: Animated Visual Effect Generation with Controllable
    Diffusion Transformer
summary: Crafting magic and illusions is one of the most thrilling aspects of
  filmmaking, with visual effects (VFX) serving as the powerhouse behind
  unforgettable cinematic experiences. While recent advances in generative
  artificial intelligence have driven progress in generic image and video
  synthesis, the domain of controllable VFX generation remains relatively
  underexplored. In this work, we propose a novel paradigm for animated VFX
  generation as image animation, where dynamic effects are generated from
  user-friendly textual descriptions and static reference images.
    Our work makes two primary contributions: (i) Open-VFX, the first
  high-quality VFX video dataset spanning 15 diverse effect categories, annotated
  with textual descriptions, instance segmentation masks for spatial
  conditioning, and start-end timestamps for temporal control. (ii) VFX Creator,
  a simple yet effective controllable VFX generation framework based on a Video
  Diffusion Transformer. The model incorporates a spatial and temporal
  controllable LoRA adapter, requiring minimal training videos. Specifically, a
  plug-and-play mask control module enables instance-level spatial manipulation,
  while tokenized start-end motion timestamps embedded in the diffusion process,
  alongside the text encoder, allow precise temporal control over effect timing
  and pace.
    Extensive experiments on the Open-VFX test set demonstrate the superiority of
  the proposed system in generating realistic and dynamic effects, achieving
  state-of-the-art performance and generalization ability in both spatial and
  temporal controllability. Furthermore, we introduce a specialized metric to
  evaluate the precision of temporal control. By bridging traditional VFX
  techniques with generative approaches, VFX Creator unlocks new possibilities
  for efficient and high-quality video effect generation, making advanced VFX
  accessible to a broader audience.
paper_authors._id: 67ab5cc2b8d7fe3b96361e31
paper_authors.name: Xinyu Liu
paper_authors._id: 67ab5cc2b8d7fe3b96361e32
paper_authors.name: Ailing Zeng
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-11T14:21:00.414Z
paper_authors._id: 665030b8c08859923b274a55
paper_authors.avatarUrl: /avatars/06a3f3cd533caa459f3da1bb5e0f4c1f.svg
paper_authors.fullname: aznukad
paper_authors.type: user
paper_authors.user: alkxncda
paper_authors._id: 67ab5cc2b8d7fe3b96361e33
paper_authors.name: Wei Xue
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-11T14:21:00.414Z
paper_authors._id: 6628adb14277eae0da5eee28
paper_authors.avatarUrl: /avatars/6cb41b80cc5e014e455dfc2a22682e64.svg
paper_authors.fullname: HKUST Audio
paper_authors.type: user
paper_authors.user: HKUST-Audio
paper_authors._id: 67ab5cc2b8d7fe3b96361e34
paper_authors.name: Harry Yang
paper_authors._id: 67ab5cc2b8d7fe3b96361e35
paper_authors.name: Wenhan Luo
paper_authors._id: 67ab5cc2b8d7fe3b96361e36
paper_authors.name: Qifeng Liu
paper_authors._id: 67ab5cc2b8d7fe3b96361e37
paper_authors.name: Yike Guo
paper_title: VFX Creator: Animated Visual Effect Generation with Controllable
    Diffusion Transformer
paper_summary: Crafting magic and illusions is one of the most thrilling aspects of
  filmmaking, with visual effects (VFX) serving as the powerhouse behind
  unforgettable cinematic experiences. While recent advances in generative
  artificial intelligence have driven progress in generic image and video
  synthesis, the domain of controllable VFX generation remains relatively
  underexplored. In this work, we propose a novel paradigm for animated VFX
  generation as image animation, where dynamic effects are generated from
  user-friendly textual descriptions and static reference images.
    Our work makes two primary contributions: (i) Open-VFX, the first
  high-quality VFX video dataset spanning 15 diverse effect categories, annotated
  with textual descriptions, instance segmentation masks for spatial
  conditioning, and start-end timestamps for temporal control. (ii) VFX Creator,
  a simple yet effective controllable VFX generation framework based on a Video
  Diffusion Transformer. The model incorporates a spatial and temporal
  controllable LoRA adapter, requiring minimal training videos. Specifically, a
  plug-and-play mask control module enables instance-level spatial manipulation,
  while tokenized start-end motion timestamps embedded in the diffusion process,
  alongside the text encoder, allow precise temporal control over effect timing
  and pace.
    Extensive experiments on the Open-VFX test set demonstrate the superiority of
  the proposed system in generating realistic and dynamic effects, achieving
  state-of-the-art performance and generalization ability in both spatial and
  temporal controllability. Furthermore, we introduce a specialized metric to
  evaluate the precision of temporal control. By bridging traditional VFX
  techniques with generative approaches, VFX Creator unlocks new possibilities
  for efficient and high-quality video effect generation, making advanced VFX
  accessible to a broader audience.
paper_ai_summary: A novel VFX generation framework, VFX Creator, uses a Video Diffusion Transformer with spatial and temporal controllable LoRA adapters to generate dynamic effects from text and images, outperforming existing methods in realism and control.
paper_ai_keywords: Video Diffusion Transformer LoRA adapter instance-level spatial manipulation tokenized start-end motion timestamps diffusion process text encoder temporal control spatial condition temporal controllability metric for temporal control