title: 4Diffusion: Multi-view Video Diffusion Model for 4D Generation
summary: Current 4D generation methods have achieved noteworthy efficacy with the aid
  of advanced diffusion generative models. However, these methods lack multi-view
  spatial-temporal modeling and encounter challenges in integrating diverse prior
  knowledge from multiple diffusion models, resulting in inconsistent temporal
  appearance and flickers. In this paper, we propose a novel 4D generation
  pipeline, namely 4Diffusion aimed at generating spatial-temporally consistent
  4D content from a monocular video. We first design a unified diffusion model
  tailored for multi-view video generation by incorporating a learnable motion
  module into a frozen 3D-aware diffusion model to capture multi-view
  spatial-temporal correlations. After training on a curated dataset, our
  diffusion model acquires reasonable temporal consistency and inherently
  preserves the generalizability and spatial consistency of the 3D-aware
  diffusion model. Subsequently, we propose 4D-aware Score Distillation Sampling
  loss, which is based on our multi-view video diffusion model, to optimize 4D
  representation parameterized by dynamic NeRF. This aims to eliminate
  discrepancies arising from multiple diffusion models, allowing for generating
  spatial-temporally consistent 4D content. Moreover, we devise an anchor loss to
  enhance the appearance details and facilitate the learning of dynamic NeRF.
  Extensive qualitative and quantitative experiments demonstrate that our method
  achieves superior performance compared to previous methods.
paper_authors._id: 665d347a6c20caf0909ccbc9
paper_authors.name: Haiyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-03T09:51:12.836Z
paper_authors._id: 645b8bf6438d6cfbe1ae47ae
paper_authors.avatarUrl: /avatars/3e4dc0523ba8e314b298567cf8f0c0ae.svg
paper_authors.fullname: Haiyu Zhang
paper_authors.type: user
paper_authors.user: aejion
paper_authors._id: 665d347a6c20caf0909ccbca
paper_authors.name: Xinyuan Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-03T09:04:38.163Z
paper_authors._id: 643e943a70c6a27621eb1c89
paper_authors.avatarUrl: /avatars/73ec521ab5ba84cc7908c52c0acef6ef.svg
paper_authors.fullname: Xinyuan Chen
paper_authors.type: user
paper_authors.user: AriaChen
paper_authors._id: 665d347a6c20caf0909ccbcb
paper_authors.name: Yaohui Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-03T09:05:01.858Z
paper_authors._id: 63201256c6b20f03c829c4b8
paper_authors.avatarUrl: /avatars/a42092119777d65e60b12eb5ba0e45f1.svg
paper_authors.fullname: Yaohui Wang
paper_authors.type: user
paper_authors.user: YaohuiW
paper_authors._id: 665d347a6c20caf0909ccbcc
paper_authors.name: Xihui Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-03T09:05:09.127Z
paper_authors._id: 65d5ec74cd05bc1eaa125040
paper_authors.avatarUrl: /avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
paper_authors.fullname: Xihui Liu
paper_authors.type: user
paper_authors.user: XihuiLiu
paper_authors._id: 665d347a6c20caf0909ccbcd
paper_authors.name: Yunhong Wang
paper_authors._id: 665d347a6c20caf0909ccbce
paper_authors.name: Yu Qiao
paper_title: 4Diffusion: Multi-view Video Diffusion Model for 4D Generation
paper_summary: Current 4D generation methods have achieved noteworthy efficacy with the aid
  of advanced diffusion generative models. However, these methods lack multi-view
  spatial-temporal modeling and encounter challenges in integrating diverse prior
  knowledge from multiple diffusion models, resulting in inconsistent temporal
  appearance and flickers. In this paper, we propose a novel 4D generation
  pipeline, namely 4Diffusion aimed at generating spatial-temporally consistent
  4D content from a monocular video. We first design a unified diffusion model
  tailored for multi-view video generation by incorporating a learnable motion
  module into a frozen 3D-aware diffusion model to capture multi-view
  spatial-temporal correlations. After training on a curated dataset, our
  diffusion model acquires reasonable temporal consistency and inherently
  preserves the generalizability and spatial consistency of the 3D-aware
  diffusion model. Subsequently, we propose 4D-aware Score Distillation Sampling
  loss, which is based on our multi-view video diffusion model, to optimize 4D
  representation parameterized by dynamic NeRF. This aims to eliminate
  discrepancies arising from multiple diffusion models, allowing for generating
  spatial-temporally consistent 4D content. Moreover, we devise an anchor loss to
  enhance the appearance details and facilitate the learning of dynamic NeRF.
  Extensive qualitative and quantitative experiments demonstrate that our method
  achieves superior performance compared to previous methods.
paper_ai_summary: A novel 4D generation pipeline, 4Diffusion, uses a unified diffusion model with a learnable motion module to generate spatial-temporally consistent 4D content from monocular video, and introduces a 4D-aware Score Distillation Sampling loss and anchor loss to improve performance.
paper_ai_keywords: diffusion generative models unified diffusion model multi-view spatial-temporal correlations frozen 3D-aware diffusion model 4D-aware Score Distillation Sampling loss dynamic NeRF anchor loss