title: Animate3D: Animating Any 3D Model with Multi-view Video Diffusion
summary: Recent advances in 4D generation mainly focus on generating 4D content by
  distilling pre-trained text or single-view image-conditioned models. It is
  inconvenient for them to take advantage of various off-the-shelf 3D assets with
  multi-view attributes, and their results suffer from spatiotemporal
  inconsistency owing to the inherent ambiguity in the supervision signals. In
  this work, we present Animate3D, a novel framework for animating any static 3D
  model. The core idea is two-fold: 1) We propose a novel multi-view video
  diffusion model (MV-VDM) conditioned on multi-view renderings of the static 3D
  object, which is trained on our presented large-scale multi-view video dataset
  (MV-Video). 2) Based on MV-VDM, we introduce a framework combining
  reconstruction and 4D Score Distillation Sampling (4D-SDS) to leverage the
  multi-view video diffusion priors for animating 3D objects. Specifically, for
  MV-VDM, we design a new spatiotemporal attention module to enhance spatial and
  temporal consistency by integrating 3D and video diffusion models.
  Additionally, we leverage the static 3D model's multi-view renderings as
  conditions to preserve its identity. For animating 3D models, an effective
  two-stage pipeline is proposed: we first reconstruct motions directly from
  generated multi-view videos, followed by the introduced 4D-SDS to refine both
  appearance and motion. Qualitative and quantitative experiments demonstrate
  that Animate3D significantly outperforms previous approaches. Data, code, and
  models will be open-released.
paper_authors._id: 669739888ee837fcda28bac8
paper_authors.name: Yanqin Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:02:30.374Z
paper_authors._id: 63738b5d3e96500368852771
paper_authors.avatarUrl: /avatars/0bf8464652751a5e355ee2a86498171f.svg
paper_authors.fullname: yanqinJiang
paper_authors.type: user
paper_authors.user: yanqinJiang
paper_authors._id: 669739888ee837fcda28bac9
paper_authors.name: Chaohui Yu
paper_authors._id: 669739888ee837fcda28baca
paper_authors.name: Chenjie Cao
paper_authors._id: 669739888ee837fcda28bacb
paper_authors.name: Fan Wang
paper_authors._id: 669739888ee837fcda28bacc
paper_authors.name: Weiming Hu
paper_authors._id: 669739888ee837fcda28bacd
paper_authors.name: Jin Gao
paper_title: Animate3D: Animating Any 3D Model with Multi-view Video Diffusion
paper_summary: Recent advances in 4D generation mainly focus on generating 4D content by
  distilling pre-trained text or single-view image-conditioned models. It is
  inconvenient for them to take advantage of various off-the-shelf 3D assets with
  multi-view attributes, and their results suffer from spatiotemporal
  inconsistency owing to the inherent ambiguity in the supervision signals. In
  this work, we present Animate3D, a novel framework for animating any static 3D
  model. The core idea is two-fold: 1) We propose a novel multi-view video
  diffusion model (MV-VDM) conditioned on multi-view renderings of the static 3D
  object, which is trained on our presented large-scale multi-view video dataset
  (MV-Video). 2) Based on MV-VDM, we introduce a framework combining
  reconstruction and 4D Score Distillation Sampling (4D-SDS) to leverage the
  multi-view video diffusion priors for animating 3D objects. Specifically, for
  MV-VDM, we design a new spatiotemporal attention module to enhance spatial and
  temporal consistency by integrating 3D and video diffusion models.
  Additionally, we leverage the static 3D model's multi-view renderings as
  conditions to preserve its identity. For animating 3D models, an effective
  two-stage pipeline is proposed: we first reconstruct motions directly from
  generated multi-view videos, followed by the introduced 4D-SDS to refine both
  appearance and motion. Qualitative and quantitative experiments demonstrate
  that Animate3D significantly outperforms previous approaches. Data, code, and
  models will be open-released.
paper_ai_keywords: multi-view video diffusion model (MV-VDM) large-scale multi-view video dataset (MV-Video) spatiotemporal attention module 3D diffusion models 4D Score Distillation Sampling (4D-SDS) multi-view renderings two-stage pipeline generated multi-view videos