Generative Rendering: Controllable 4D-Guided Video Generation with 2D
  Diffusion Models
Traditional 3D content creation tools empower users to bring their
imagination to life by giving them direct control over a scene's geometry,
appearance, motion, and camera path. Creating computer-generated videos,
however, is a tedious manual process, which can be automated by emerging
text-to-video diffusion models. Despite great promise, video diffusion models
are difficult to control, hindering a user to apply their own creativity rather
than amplifying it. To address this challenge, we present a novel approach that
combines the controllability of dynamic 3D meshes with the expressivity and
editability of emerging diffusion models. For this purpose, our approach takes
an animated, low-fidelity rendered mesh as input and injects the ground truth
correspondence information obtained from the dynamic mesh into various stages
of a pre-trained text-to-image generation model to output high-quality and
temporally consistent frames. We demonstrate our approach on various examples
where motion can be obtained by animating rigged assets or changing the camera
path.
656e8d822cd7a3e348fe00c0
Shengqu Cai
claimed_verified
2024-11-28T09:20:30.203Z
66a1da7cc9e703d2af5ad742
/avatars/f9cd9ae3407e249ab4569479200feb1f.svg
Shengqu Cai
user
primecai
656e8d822cd7a3e348fe00c1
Duygu Ceylan
656e8d822cd7a3e348fe00c2
Matheus Gadelha
admin_assigned
2023-12-05T12:09:07.528Z
6315a9c466062476ae1fa0c7
/avatars/c59749be7ecee4858c59889fea61c88a.svg
Matheus
user
matheusgadelha
656e8d822cd7a3e348fe00c3
Chun-Hao Paul Huang
admin_assigned
2023-12-05T12:09:19.593Z
6318069a33a4e2b5d3b47fe5
/avatars/c9a9818e64186746557466cc7a6d57b8.svg
CHUN-HAO  HUANG
user
ccool2486
656e8d822cd7a3e348fe00c4
Tuanfeng Yang Wang
admin_assigned
2023-12-05T12:09:42.306Z
630403daeedc089484c76b61
/avatars/30bd8d24dbf88ceb6ca823618ac47d95.svg
tuanfeng wang
user
yangtwan
656e8d822cd7a3e348fe00c5
Gordon Wetzstein
Generative Rendering: Controllable 4D-Guided Video Generation with 2D
  Diffusion Models
Traditional 3D content creation tools empower users to bring their
imagination to life by giving them direct control over a scene's geometry,
appearance, motion, and camera path. Creating computer-generated videos,
however, is a tedious manual process, which can be automated by emerging
text-to-video diffusion models. Despite great promise, video diffusion models
are difficult to control, hindering a user to apply their own creativity rather
than amplifying it. To address this challenge, we present a novel approach that
combines the controllability of dynamic 3D meshes with the expressivity and
editability of emerging diffusion models. For this purpose, our approach takes
an animated, low-fidelity rendered mesh as input and injects the ground truth
correspondence information obtained from the dynamic mesh into various stages
of a pre-trained text-to-image generation model to output high-quality and
temporally consistent frames. We demonstrate our approach on various examples
where motion can be obtained by animating rigged assets or changing the camera
path.
The novel approach combines dynamic 3D mesh controllability with text-to-image diffusion models to generate high-quality, temporally consistent video frames from low-fidelity inputs.
3D mesh diffusion models text-to-image generation ground truth correspondence rigged assets camera path