title: AutoVFX: Physically Realistic Video Editing from Natural Language
    Instructions
summary: Modern visual effects (VFX) software has made it possible for skilled artists
  to create imagery of virtually anything. However, the creation process remains
  laborious, complex, and largely inaccessible to everyday users. In this work,
  we present AutoVFX, a framework that automatically creates realistic and
  dynamic VFX videos from a single video and natural language instructions. By
  carefully integrating neural scene modeling, LLM-based code generation, and
  physical simulation, AutoVFX is able to provide physically-grounded,
  photorealistic editing effects that can be controlled directly using natural
  language instructions. We conduct extensive experiments to validate AutoVFX's
  efficacy across a diverse spectrum of videos and instructions. Quantitative and
  qualitative results suggest that AutoVFX outperforms all competing methods by a
  large margin in generative quality, instruction alignment, editing versatility,
  and physical plausibility.
paper_authors._id: 672a50c1cd1b2e694cad25e6
paper_authors.name: Hao-Yu Hsu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T20:17:28.523Z
paper_authors._id: 652770b2c99126d30f939a19
paper_authors.avatarUrl: /avatars/8f0bf756079ccb61c7cde323a4b19748.svg
paper_authors.fullname: Hao-Yu Hsu
paper_authors.type: user
paper_authors.user: haoyuhsu
paper_authors._id: 672a50c1cd1b2e694cad25e7
paper_authors.name: Zhi-Hao Lin
paper_authors._id: 672a50c1cd1b2e694cad25e8
paper_authors.name: Albert Zhai
paper_authors._id: 672a50c1cd1b2e694cad25e9
paper_authors.name: Hongchi Xia
paper_authors._id: 672a50c1cd1b2e694cad25ea
paper_authors.name: Shenlong Wang
paper_title: AutoVFX: Physically Realistic Video Editing from Natural Language
    Instructions
paper_summary: Modern visual effects (VFX) software has made it possible for skilled artists
  to create imagery of virtually anything. However, the creation process remains
  laborious, complex, and largely inaccessible to everyday users. In this work,
  we present AutoVFX, a framework that automatically creates realistic and
  dynamic VFX videos from a single video and natural language instructions. By
  carefully integrating neural scene modeling, LLM-based code generation, and
  physical simulation, AutoVFX is able to provide physically-grounded,
  photorealistic editing effects that can be controlled directly using natural
  language instructions. We conduct extensive experiments to validate AutoVFX's
  efficacy across a diverse spectrum of videos and instructions. Quantitative and
  qualitative results suggest that AutoVFX outperforms all competing methods by a
  large margin in generative quality, instruction alignment, editing versatility,
  and physical plausibility.
paper_ai_keywords: neural scene modeling LLM-based code generation physical simulation physically-grounded photorealistic generative quality instruction alignment editing versatility physical plausibility