title: Generative Rendering: Controllable 4D-Guided Video Generation with 2D
    Diffusion Models
summary: Traditional 3D content creation tools empower users to bring their
  imagination to life by giving them direct control over a scene's geometry,
  appearance, motion, and camera path. Creating computer-generated videos,
  however, is a tedious manual process, which can be automated by emerging
  text-to-video diffusion models. Despite great promise, video diffusion models
  are difficult to control, hindering a user to apply their own creativity rather
  than amplifying it. To address this challenge, we present a novel approach that
  combines the controllability of dynamic 3D meshes with the expressivity and
  editability of emerging diffusion models. For this purpose, our approach takes
  an animated, low-fidelity rendered mesh as input and injects the ground truth
  correspondence information obtained from the dynamic mesh into various stages
  of a pre-trained text-to-image generation model to output high-quality and
  temporally consistent frames. We demonstrate our approach on various examples
  where motion can be obtained by animating rigged assets or changing the camera
  path.
paper_authors._id: 656e8d822cd7a3e348fe00c0
paper_authors.name: Shengqu Cai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-28T09:20:30.203Z
paper_authors._id: 66a1da7cc9e703d2af5ad742
paper_authors.avatarUrl: /avatars/f9cd9ae3407e249ab4569479200feb1f.svg
paper_authors.fullname: Shengqu Cai
paper_authors.type: user
paper_authors.user: primecai
paper_authors._id: 656e8d822cd7a3e348fe00c1
paper_authors.name: Duygu Ceylan
paper_authors._id: 656e8d822cd7a3e348fe00c2
paper_authors.name: Matheus Gadelha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T12:09:07.528Z
paper_authors._id: 6315a9c466062476ae1fa0c7
paper_authors.avatarUrl: /avatars/c59749be7ecee4858c59889fea61c88a.svg
paper_authors.fullname: Matheus
paper_authors.type: user
paper_authors.user: matheusgadelha
paper_authors._id: 656e8d822cd7a3e348fe00c3
paper_authors.name: Chun-Hao Paul Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T12:09:19.593Z
paper_authors._id: 6318069a33a4e2b5d3b47fe5
paper_authors.avatarUrl: /avatars/c9a9818e64186746557466cc7a6d57b8.svg
paper_authors.fullname: CHUN-HAO  HUANG
paper_authors.type: user
paper_authors.user: ccool2486
paper_authors._id: 656e8d822cd7a3e348fe00c4
paper_authors.name: Tuanfeng Yang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T12:09:42.306Z
paper_authors._id: 630403daeedc089484c76b61
paper_authors.avatarUrl: /avatars/30bd8d24dbf88ceb6ca823618ac47d95.svg
paper_authors.fullname: tuanfeng wang
paper_authors.type: user
paper_authors.user: yangtwan
paper_authors._id: 656e8d822cd7a3e348fe00c5
paper_authors.name: Gordon Wetzstein
paper_title: Generative Rendering: Controllable 4D-Guided Video Generation with 2D
    Diffusion Models
paper_summary: Traditional 3D content creation tools empower users to bring their
  imagination to life by giving them direct control over a scene's geometry,
  appearance, motion, and camera path. Creating computer-generated videos,
  however, is a tedious manual process, which can be automated by emerging
  text-to-video diffusion models. Despite great promise, video diffusion models
  are difficult to control, hindering a user to apply their own creativity rather
  than amplifying it. To address this challenge, we present a novel approach that
  combines the controllability of dynamic 3D meshes with the expressivity and
  editability of emerging diffusion models. For this purpose, our approach takes
  an animated, low-fidelity rendered mesh as input and injects the ground truth
  correspondence information obtained from the dynamic mesh into various stages
  of a pre-trained text-to-image generation model to output high-quality and
  temporally consistent frames. We demonstrate our approach on various examples
  where motion can be obtained by animating rigged assets or changing the camera
  path.
paper_ai_keywords: dynamic 3D meshes text-to-video diffusion models bit truth correspondence information pre-trained text-to-image generation model temporally consistent frames rigged assets