title: Bringing Objects to Life: 4D generation from 3D objects
summary: Recent advancements in generative modeling now enable the creation of 4D
  content (moving 3D objects) controlled with text prompts. 4D generation has
  large potential in applications like virtual worlds, media, and gaming, but
  existing methods provide limited control over the appearance and geometry of
  generated content. In this work, we introduce a method for animating
  user-provided 3D objects by conditioning on textual prompts to guide 4D
  generation, enabling custom animations while maintaining the identity of the
  original object. We first convert a 3D mesh into a ``static" 4D Neural Radiance
  Field (NeRF) that preserves the visual attributes of the input object. Then, we
  animate the object using an Image-to-Video diffusion model driven by text. To
  improve motion realism, we introduce an incremental viewpoint selection
  protocol for sampling perspectives to promote lifelike movement and a masked
  Score Distillation Sampling (SDS) loss, which leverages attention maps to focus
  optimization on relevant regions. We evaluate our model in terms of temporal
  coherence, prompt adherence, and visual fidelity and find that our method
  outperforms baselines that are based on other approaches, achieving up to
  threefold improvements in identity preservation measured using LPIPS scores,
  and effectively balancing visual quality with dynamic content.
paper_authors._id: 67738c503be2064a656e7e70
paper_authors.name: Ohad Rahamim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-01T20:12:39.503Z
paper_authors._id: 63c59c3a6d132b995fedface
paper_authors.avatarUrl: /avatars/4e18b19e477cb683ce1ba3ae6ab77d8e.svg
paper_authors.fullname: Ohad rahamim
paper_authors.type: user
paper_authors.user: ohad204
paper_authors._id: 67738c503be2064a656e7e71
paper_authors.name: Ori Malca
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-01T20:12:41.929Z
paper_authors._id: 643ee7c8947f8be48fcb02c0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643ee7c8947f8be48fcb02c0/BOTz46LpyOZiC9BfFG_Pn.png
paper_authors.fullname: Ori Malca
paper_authors.type: user
paper_authors.user: Orimalca
paper_authors._id: 67738c503be2064a656e7e72
paper_authors.name: Dvir Samuel
paper_authors._id: 67738c503be2064a656e7e73
paper_authors.name: Gal Chechik
paper_title: Bringing Objects to Life: 4D generation from 3D objects
paper_summary: Recent advancements in generative modeling now enable the creation of 4D
  content (moving 3D objects) controlled with text prompts. 4D generation has
  large potential in applications like virtual worlds, media, and gaming, but
  existing methods provide limited control over the appearance and geometry of
  generated content. In this work, we introduce a method for animating
  user-provided 3D objects by conditioning on textual prompts to guide 4D
  generation, enabling custom animations while maintaining the identity of the
  original object. We first convert a 3D mesh into a ``static" 4D Neural Radiance
  Field (NeRF) that preserves the visual attributes of the input object. Then, we
  animate the object using an Image-to-Video diffusion model driven by text. To
  improve motion realism, we introduce an incremental viewpoint selection
  protocol for sampling perspectives to promote lifelike movement and a masked
  Score Distillation Sampling (SDS) loss, which leverages attention maps to focus
  optimization on relevant regions. We evaluate our model in terms of temporal
  coherence, prompt adherence, and visual fidelity and find that our method
  outperforms baselines that are based on other approaches, achieving up to
  threefold improvements in identity preservation measured using LPIPS scores,
  and effectively balancing visual quality with dynamic content.
paper_ai_keywords: Generative modeling 4D objects 3D objects NeRF (Neural Radiance Field) diffusion model verbal perspectives completeness selection protocol Score Distillation factor (SDS) attention maps temporal coherence prompt adherence visual fidelity LPIPS scores dynamic content