Bringing Objects to Life: 4D generation from 3D objects
Recent advancements in generative modeling now enable the creation of 4D
content (moving 3D objects) controlled with text prompts. 4D generation has
large potential in applications like virtual worlds, media, and gaming, but
existing methods provide limited control over the appearance and geometry of
generated content. In this work, we introduce a method for animating
user-provided 3D objects by conditioning on textual prompts to guide 4D
generation, enabling custom animations while maintaining the identity of the
original object. We first convert a 3D mesh into a ``static" 4D Neural Radiance
Field (NeRF) that preserves the visual attributes of the input object. Then, we
animate the object using an Image-to-Video diffusion model driven by text. To
improve motion realism, we introduce an incremental viewpoint selection
protocol for sampling perspectives to promote lifelike movement and a masked
Score Distillation Sampling (SDS) loss, which leverages attention maps to focus
optimization on relevant regions. We evaluate our model in terms of temporal
coherence, prompt adherence, and visual fidelity and find that our method
outperforms baselines that are based on other approaches, achieving up to
threefold improvements in identity preservation measured using LPIPS scores,
and effectively balancing visual quality with dynamic content.
67738c503be2064a656e7e70
Ohad Rahamim
claimed_verified
2025-01-01T20:12:39.503Z
63c59c3a6d132b995fedface
/avatars/4e18b19e477cb683ce1ba3ae6ab77d8e.svg
Ohad rahamim
user
ohad204
67738c503be2064a656e7e71
Ori Malca
claimed_verified
2025-01-01T20:12:41.929Z
643ee7c8947f8be48fcb02c0
https://cdn-avatars.huggingface.co/v1/production/uploads/643ee7c8947f8be48fcb02c0/BOTz46LpyOZiC9BfFG_Pn.png
Ori Malca
user
Orimalca
67738c503be2064a656e7e72
Dvir Samuel
claimed_verified
2025-03-25T09:05:47.986Z
630f0d48982455e61cc4cc08
/avatars/eea6ed2e112e830effa98a4661c5474f.svg
Samuel
user
Dvir
67738c503be2064a656e7e73
Gal Chechik
Bringing Objects to Life: 4D generation from 3D objects
Recent advancements in generative modeling now enable the creation of 4D
content (moving 3D objects) controlled with text prompts. 4D generation has
large potential in applications like virtual worlds, media, and gaming, but
existing methods provide limited control over the appearance and geometry of
generated content. In this work, we introduce a method for animating
user-provided 3D objects by conditioning on textual prompts to guide 4D
generation, enabling custom animations while maintaining the identity of the
original object. We first convert a 3D mesh into a ``static" 4D Neural Radiance
Field (NeRF) that preserves the visual attributes of the input object. Then, we
animate the object using an Image-to-Video diffusion model driven by text. To
improve motion realism, we introduce an incremental viewpoint selection
protocol for sampling perspectives to promote lifelike movement and a masked
Score Distillation Sampling (SDS) loss, which leverages attention maps to focus
optimization on relevant regions. We evaluate our model in terms of temporal
coherence, prompt adherence, and visual fidelity and find that our method
outperforms baselines that are based on other approaches, achieving up to
threefold improvements in identity preservation measured using LPIPS scores,
and effectively balancing visual quality with dynamic content.
A method for animating 3D objects using text prompts to guide 4D generation, employing Neural Radiance Fields and diffusion models, achieves high identity preservation and visual quality.
Neural Radiance Field (NeRF) Image-to-Video diffusion model incremental viewpoint selection masked Score Distillation Sampling (SDS) loss attention maps LPIPS scores