title: Consistent4D: Consistent 360° Dynamic Object Generation from
    Monocular Video
summary: In this paper, we present Consistent4D, a novel approach for generating 4D
  dynamic objects from uncalibrated monocular videos. Uniquely, we cast the
  360-degree dynamic object reconstruction as a 4D generation problem,
  eliminating the need for tedious multi-view data collection and camera
  calibration. This is achieved by leveraging the object-level 3D-aware image
  diffusion model as the primary supervision signal for training Dynamic Neural
  Radiance Fields (DyNeRF). Specifically, we propose a Cascade DyNeRF to
  facilitate stable convergence and temporal continuity under the supervision
  signal which is discrete along the time axis. To achieve spatial and temporal
  consistency, we further introduce an Interpolation-driven Consistency Loss. It
  is optimized by minimizing the discrepancy between rendered frames from DyNeRF
  and interpolated frames from a pre-trained video interpolation model. Extensive
  experiments show that our Consistent4D can perform competitively to prior art
  alternatives, opening up new possibilities for 4D dynamic object generation
  from monocular videos, whilst also demonstrating advantage for conventional
  text-to-3D generation tasks. Our project page is
  https://consistent4d.github.io/.
paper_authors._id: 6549e4954aecf5964cc29c66
paper_authors.name: Yanqin Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-08T07:57:29.553Z
paper_authors._id: 63738b5d3e96500368852771
paper_authors.avatarUrl: /avatars/0bf8464652751a5e355ee2a86498171f.svg
paper_authors.fullname: yanqinJiang
paper_authors.type: user
paper_authors.user: yanqinJiang
paper_authors._id: 6549e4954aecf5964cc29c67
paper_authors.name: Li Zhang
paper_authors._id: 6549e4954aecf5964cc29c68
paper_authors.name: Jin Gao
paper_authors._id: 6549e4954aecf5964cc29c69
paper_authors.name: Weimin Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T16:29:15.205Z
paper_authors._id: 64f69d075207c759d09a5b6f
paper_authors.avatarUrl: /avatars/9c4ce43746bc89f51e41ce220a60aad8.svg
paper_authors.fullname: huweiming
paper_authors.type: user
paper_authors.user: huweiming
paper_authors._id: 6549e4954aecf5964cc29c6a
paper_authors.name: Yao Yao
paper_title: Consistent4D: Consistent 360° Dynamic Object Generation from
    Monocular Video
paper_summary: In this paper, we present Consistent4D, a novel approach for generating 4D
  dynamic objects from uncalibrated monocular videos. Uniquely, we cast the
  360-degree dynamic object reconstruction as a 4D generation problem,
  eliminating the need for tedious multi-view data collection and camera
  calibration. This is achieved by leveraging the object-level 3D-aware image
  diffusion model as the primary supervision signal for training Dynamic Neural
  Radiance Fields (DyNeRF). Specifically, we propose a Cascade DyNeRF to
  facilitate stable convergence and temporal continuity under the supervision
  signal which is discrete along the time axis. To achieve spatial and temporal
  consistency, we further introduce an Interpolation-driven Consistency Loss. It
  is optimized by minimizing the discrepancy between rendered frames from DyNeRF
  and interpolated frames from a pre-trained video interpolation model. Extensive
  experiments show that our Consistent4D can perform competitively to prior art
  alternatives, opening up new possibilities for 4D dynamic object generation
  from monocular videos, whilst also demonstrating advantage for conventional
  text-to-3D generation tasks. Our project page is
  https://consistent4d.github.io/.
paper_ai_summary: Consistent4D uses a 3D-aware image diffusion model and Cascade DyNeRF to generate 4D dynamic objects from uncalibrated monocular videos, improving spatial and temporal consistency with an Interpolation-driven Consistency Loss.
paper_ai_keywords: 3D-aware image diffusion model Dynamic Neural Radiance Fields DyNeRF Cascade DyNeRF Interpolation-driven Consistency Loss