title: DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction
    Model
summary: We propose DMV3D, a novel 3D generation approach that uses a
  transformer-based 3D large reconstruction model to denoise multi-view
  diffusion. Our reconstruction model incorporates a triplane NeRF representation
  and can denoise noisy multi-view images via NeRF reconstruction and rendering,
  achieving single-stage 3D generation in sim30s on single A100 GPU. We train
  DMV3D on large-scale multi-view image datasets of highly diverse
  objects using only image reconstruction losses, without accessing 3D assets. We
  demonstrate state-of-the-art results for the single-image reconstruction
  problem where probabilistic modeling of unseen object parts is required for
  generating diverse reconstructions with sharp textures. We also show
  high-quality text-to-3D generation results outperforming previous 3D diffusion
  models. Our project website is at: https://justimyhxu.github.io/projects/dmv3d/ .
paper_authors._id: 6555ad0bf96492362e33e564
paper_authors.name: Yinghao Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-01T09:28:21.740Z
paper_authors._id: 6555da9adcf410fd0753569c
paper_authors.avatarUrl: /avatars/ac58a796bb54f334fdb475a4b75c4d27.svg
paper_authors.fullname: Yinghao Xu
paper_authors.type: user
paper_authors.user: justimyhxu
paper_authors._id: 6555ad0bf96492362e33e565
paper_authors.name: Hao Tan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-16T08:30:14.956Z
paper_authors._id: 5f297389e923d665e61627f9
paper_authors.avatarUrl: /avatars/c39652c5b9ee182f4ea4dcd2f528f61b.svg
paper_authors.fullname: Hao Tan
paper_authors.type: user
paper_authors.user: airsplay
paper_authors._id: 6555ad0bf96492362e33e566
paper_authors.name: Fujun Luan
paper_authors._id: 6555ad0bf96492362e33e567
paper_authors.name: Sai Bi
paper_authors._id: 6555ad0bf96492362e33e568
paper_authors.name: Peng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-16T08:31:33.365Z
paper_authors._id: 6486ad27d29294981d2a7ebd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/w-PUHQzMikeCU4matUio_.png
paper_authors.fullname: Peng Wang
paper_authors.type: user
paper_authors.user: totoro97
paper_authors._id: 6555ad0bf96492362e33e569
paper_authors.name: Jiahao Li
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-11-16T05:47:59.116Z
paper_authors._id: 630bffcf8b327c7b8b9dfcce
paper_authors.avatarUrl: /avatars/12691f869f01217f7b92a326c30716b1.svg
paper_authors.fullname: Jiahao Li
paper_authors.type: user
paper_authors.user: 54jiahao
paper_authors._id: 6555ad0bf96492362e33e56a
paper_authors.name: Zifan Shi
paper_authors._id: 6555ad0bf96492362e33e56b
paper_authors.name: Kalyan Sunkavalli
paper_authors._id: 6555ad0bf96492362e33e56c
paper_authors.name: Gordon Wetzstein
paper_authors._id: 6555ad0bf96492362e33e56d
paper_authors.name: Zexiang Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-16T20:01:33.141Z
paper_authors._id: 65565b1b8616d8d740abb976
paper_authors.avatarUrl: /avatars/c245b19ed05292bf1bf4a9d8cf2a409c.svg
paper_authors.fullname: Zexiang Xu
paper_authors.type: user
paper_authors.user: zexiangxu
paper_authors._id: 6555ad0bf96492362e33e56e
paper_authors.name: Kai Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-16T08:33:10.095Z
paper_authors._id: 6351a882a4845f26d16a28d7
paper_authors.avatarUrl: /avatars/4f8629fbf22faeb6511f4068f2828809.svg
paper_authors.fullname: Kai Zhang
paper_authors.type: user
paper_authors.user: Kai-46
paper_title: DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction
    Model
paper_summary: We propose DMV3D, a novel 3D generation approach that uses a
  transformer-based 3D large reconstruction model to denoise multi-view
  diffusion. Our reconstruction model incorporates a triplane NeRF representation
  and can denoise noisy multi-view images via NeRF reconstruction and rendering,
  achieving single-stage 3D generation in sim30s on single A100 GPU. We train
  DMV3D on large-scale multi-view image datasets of highly diverse
  objects using only image reconstruction losses, without accessing 3D assets. We
  demonstrate state-of-the-art results for the single-image reconstruction
  problem where probabilistic modeling of unseen object parts is required for
  generating diverse reconstructions with sharp textures. We also show
  high-quality text-to-3D generation results outperforming previous 3D diffusion
  models. Our project website is at: https://justimyhxu.github.io/projects/dmv3d/ .
paper_ai_keywords: Transformer-based Denoising Multi-view diffusion NeRF (Neural Radiance Fields) Triplanes Single-stage 3D generation Image reconstruction losses Probabilistic modeling Text-to-3D generation