AutoVFX: Physically Realistic Video Editing from Natural Language
  Instructions
Modern visual effects (VFX) software has made it possible for skilled artists
to create imagery of virtually anything. However, the creation process remains
laborious, complex, and largely inaccessible to everyday users. In this work,
we present AutoVFX, a framework that automatically creates realistic and
dynamic VFX videos from a single video and natural language instructions. By
carefully integrating neural scene modeling, LLM-based code generation, and
physical simulation, AutoVFX is able to provide physically-grounded,
photorealistic editing effects that can be controlled directly using natural
language instructions. We conduct extensive experiments to validate AutoVFX's
efficacy across a diverse spectrum of videos and instructions. Quantitative and
qualitative results suggest that AutoVFX outperforms all competing methods by a
large margin in generative quality, instruction alignment, editing versatility,
and physical plausibility.
672a50c1cd1b2e694cad25e6
Hao-Yu Hsu
claimed_verified
2024-11-05T20:17:28.523Z
652770b2c99126d30f939a19
/avatars/8f0bf756079ccb61c7cde323a4b19748.svg
Hao-Yu Hsu
user
haoyuhsu
672a50c1cd1b2e694cad25e7
Zhi-Hao Lin
672a50c1cd1b2e694cad25e8
Albert Zhai
672a50c1cd1b2e694cad25e9
Hongchi Xia
672a50c1cd1b2e694cad25ea
Shenlong Wang
AutoVFX: Physically Realistic Video Editing from Natural Language
  Instructions
Modern visual effects (VFX) software has made it possible for skilled artists
to create imagery of virtually anything. However, the creation process remains
laborious, complex, and largely inaccessible to everyday users. In this work,
we present AutoVFX, a framework that automatically creates realistic and
dynamic VFX videos from a single video and natural language instructions. By
carefully integrating neural scene modeling, LLM-based code generation, and
physical simulation, AutoVFX is able to provide physically-grounded,
photorealistic editing effects that can be controlled directly using natural
language instructions. We conduct extensive experiments to validate AutoVFX's
efficacy across a diverse spectrum of videos and instructions. Quantitative and
qualitative results suggest that AutoVFX outperforms all competing methods by a
large margin in generative quality, instruction alignment, editing versatility,
and physical plausibility.
AutoVFX generates realistic VFX videos from a single video and natural language instructions using neural scene modeling, LLM-based code generation, and physical simulation.
neural scene modeling LLM-based code generation physical simulation