LN3Diff: Scalable Latent Neural Fields Diffusion for Speedy 3D
  Generation
The field of neural rendering has witnessed significant progress with
advancements in generative models and differentiable rendering techniques.
Though 2D diffusion has achieved success, a unified 3D diffusion pipeline
remains unsettled. This paper introduces a novel framework called LN3Diff to
address this gap and enable fast, high-quality, and generic conditional 3D
generation. Our approach harnesses a 3D-aware architecture and variational
autoencoder (VAE) to encode the input image into a structured, compact, and 3D
latent space. The latent is decoded by a transformer-based decoder into a
high-capacity 3D neural field. Through training a diffusion model on this
3D-aware latent space, our method achieves state-of-the-art performance on
ShapeNet for 3D generation and demonstrates superior performance in monocular
3D reconstruction and conditional 3D generation across various datasets.
Moreover, it surpasses existing 3D diffusion methods in terms of inference
speed, requiring no per-instance optimization. Our proposed LN3Diff presents a
significant advancement in 3D generative modeling and holds promise for various
applications in 3D vision and graphics tasks.
65f90de4a5051f13e5e04f62
Yushi Lan
claimed_verified
2024-03-19T14:31:28.088Z
64b55f62e6b0c53f97e64034
/avatars/d5fc53de37d4fb0b5095a5fb21b43503.svg
LAN YUSHI
user
yslan
65f90de4a5051f13e5e04f63
Fangzhou Hong
admin_assigned
2024-03-19T10:06:19.390Z
623c530013a63ea865f96c8e
/avatars/164455a1a94f92b71733fc778c21bd89.svg
Fangzhou Hong
user
hongfz16
65f90de4a5051f13e5e04f64
Shuai Yang
admin_assigned
2024-03-19T10:06:43.628Z
62a54d0410334c1d024e2f59
https://cdn-avatars.huggingface.co/v1/production/uploads/1664764278226-62a54d0410334c1d024e2f59.jpeg
Shuai Yang
user
PKUWilliamYang
65f90de4a5051f13e5e04f65
Shangchen Zhou
admin_assigned
2024-03-19T10:06:54.500Z
62e57662ae9d3f10acbb1b1b
https://cdn-avatars.huggingface.co/v1/production/uploads/62e57662ae9d3f10acbb1b1b/lg58jdbNyv6LGH2LFnZDF.png
Shangchen Zhou
user
sczhou
65f90de4a5051f13e5e04f66
Xuyi Meng
65f90de4a5051f13e5e04f67
Bo Dai
65f90de4a5051f13e5e04f68
Xingang Pan
admin_assigned
2024-03-19T10:08:01.688Z
646758d6a9b4610868a138ff
https://cdn-avatars.huggingface.co/v1/production/uploads/646758d6a9b4610868a138ff/fNBlK5g4g7vlPAR1bQimZ.png
Xingang Pan
user
Xingang-Pan
65f90de4a5051f13e5e04f69
Chen Change Loy
LN3Diff: Scalable Latent Neural Fields Diffusion for Speedy 3D
  Generation
The field of neural rendering has witnessed significant progress with
advancements in generative models and differentiable rendering techniques.
Though 2D diffusion has achieved success, a unified 3D diffusion pipeline
remains unsettled. This paper introduces a novel framework called LN3Diff to
address this gap and enable fast, high-quality, and generic conditional 3D
generation. Our approach harnesses a 3D-aware architecture and variational
autoencoder (VAE) to encode the input image into a structured, compact, and 3D
latent space. The latent is decoded by a transformer-based decoder into a
high-capacity 3D neural field. Through training a diffusion model on this
3D-aware latent space, our method achieves state-of-the-art performance on
ShapeNet for 3D generation and demonstrates superior performance in monocular
3D reconstruction and conditional 3D generation across various datasets.
Moreover, it surpasses existing 3D diffusion methods in terms of inference
speed, requiring no per-instance optimization. Our proposed LN3Diff presents a
significant advancement in 3D generative modeling and holds promise for various
applications in 3D vision and graphics tasks.
A novel 3D diffusion framework, LN3Diff, uses a 3D-aware architecture and VAE to enable fast, high-quality 3D generation, outperforming existing methods in reconstruction speed and quality on various datasets.
generative models differentiable rendering techniques 2D diffusion unified 3D diffusion pipeline LN3Diff 3D-aware architecture variational autoencoder VAE transformer-based decoder 3D neural field ShapeNet monocular 3D reconstruction conditional 3D generation