title: LN3Diff: Scalable Latent Neural Fields Diffusion for Speedy 3D
    Generation
summary: The field of neural rendering has witnessed significant progress with
  advancements in generative models and differentiable rendering techniques.
  Though 2D diffusion has achieved success, a unified 3D diffusion pipeline
  remains unsettled. This paper introduces a novel framework called LN3Diff to
  address this gap and enable fast, high-quality, and generic conditional 3D
  generation. Our approach harnesses a 3D-aware architecture and variational
  autoencoder (VAE) to encode the input image into a structured, compact, and 3D
  latent space. The latent is decoded by a transformer-based decoder into a
  high-capacity 3D neural field. Through training a diffusion model on this
  3D-aware latent space, our method achieves state-of-the-art performance on
  ShapeNet for 3D generation and demonstrates superior performance in monocular
  3D reconstruction and conditional 3D generation across various datasets.
  Moreover, it surpasses existing 3D diffusion methods in terms of inference
  speed, requiring no per-instance optimization. Our proposed LN3Diff presents a
  significant advancement in 3D generative modeling and holds promise for various
  applications in 3D vision and graphics tasks.
paper_authors._id: 65f90de4a5051f13e5e04f62
paper_authors.name: Yushi Lan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-19T14:31:28.088Z
paper_authors._id: 64b55f62e6b0c53f97e64034
paper_authors.avatarUrl: /avatars/d5fc53de37d4fb0b5095a5fb21b43503.svg
paper_authors.fullname: LAN YUSHI
paper_authors.type: user
paper_authors.user: yslan
paper_authors._id: 65f90de4a5051f13e5e04f63
paper_authors.name: Fangzhou Hong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T10:06:19.390Z
paper_authors._id: 623c530013a63ea865f96c8e
paper_authors.avatarUrl: /avatars/164455a1a94f92b71733fc778c21bd89.svg
paper_authors.fullname: Fangzhou Hong
paper_authors.type: user
paper_authors.user: hongfz16
paper_authors._id: 65f90de4a5051f13e5e04f64
paper_authors.name: Shuai Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T10:06:43.628Z
paper_authors._id: 62a54d0410334c1d024e2f59
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664764278226-62a54d0410334c1d024e2f59.jpeg
paper_authors.fullname: Shuai Yang
paper_authors.type: user
paper_authors.user: PKUWilliamYang
paper_authors._id: 65f90de4a5051f13e5e04f65
paper_authors.name: Shangchen Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T10:06:54.500Z
paper_authors._id: 62e57662ae9d3f10acbb1b1b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62e57662ae9d3f10acbb1b1b/lg58jdbNyv6LGH2LFnZDF.png
paper_authors.fullname: Shangchen Zhou
paper_authors.type: user
paper_authors.user: sczhou
paper_authors._id: 65f90de4a5051f13e5e04f66
paper_authors.name: Xuyi Meng
paper_authors._id: 65f90de4a5051f13e5e04f67
paper_authors.name: Bo Dai
paper_authors._id: 65f90de4a5051f13e5e04f68
paper_authors.name: Xingang Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T10:08:01.688Z
paper_authors._id: 646758d6a9b4610868a138ff
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646758d6a9b4610868a138ff/fNBlK5g4g7vlPAR1bQimZ.png
paper_authors.fullname: Xingang Pan
paper_authors.type: user
paper_authors.user: Xingang-Pan
paper_authors._id: 65f90de4a5051f13e5e04f69
paper_authors.name: Chen Change Loy
paper_title: LN3Diff: Scalable Latent Neural Fields Diffusion for Speedy 3D
    Generation
paper_summary: The field of neural rendering has witnessed significant progress with
  advancements in generative models and differentiable rendering techniques.
  Though 2D diffusion has achieved success, a unified 3D diffusion pipeline
  remains unsettled. This paper introduces a novel framework called LN3Diff to
  address this gap and enable fast, high-quality, and generic conditional 3D
  generation. Our approach harnesses a 3D-aware architecture and variational
  autoencoder (VAE) to encode the input image into a structured, compact, and 3D
  latent space. The latent is decoded by a transformer-based decoder into a
  high-capacity 3D neural field. Through training a diffusion model on this
  3D-aware latent space, our method achieves state-of-the-art performance on
  ShapeNet for 3D generation and demonstrates superior performance in monocular
  3D reconstruction and conditional 3D generation across various datasets.
  Moreover, it surpasses existing 3D diffusion methods in terms of inference
  speed, requiring no per-instance optimization. Our proposed LN3Diff presents a
  significant advancement in 3D generative modeling and holds promise for various
  applications in 3D vision and graphics tasks.
paper_ai_summary: A novel 3D diffusion framework, LN3Diff, uses a 3D-aware architecture and VAE to enable fast, high-quality 3D generation, outperforming existing methods in reconstruction speed and quality on various datasets.
paper_ai_keywords: generative models differentiable rendering techniques 2D diffusion unified 3D diffusion pipeline LN3Diff 3D-aware architecture variational autoencoder VAE transformer-based decoder 3D neural field ShapeNet monocular 3D reconstruction conditional 3D generation