BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane
  Extrapolation
We present BlockFusion, a diffusion-based model that generates 3D scenes as
unit blocks and seamlessly incorporates new blocks to extend the scene.
BlockFusion is trained using datasets of 3D blocks that are randomly cropped
from complete 3D scene meshes. Through per-block fitting, all training blocks
are converted into the hybrid neural fields: with a tri-plane containing the
geometry features, followed by a Multi-layer Perceptron (MLP) for decoding the
signed distance values. A variational auto-encoder is employed to compress the
tri-planes into the latent tri-plane space, on which the denoising diffusion
process is performed. Diffusion applied to the latent representations allows
for high-quality and diverse 3D scene generation. To expand a scene during
generation, one needs only to append empty blocks to overlap with the current
scene and extrapolate existing latent tri-planes to populate new blocks. The
extrapolation is done by conditioning the generation process with the feature
samples from the overlapping tri-planes during the denoising iterations. Latent
tri-plane extrapolation produces semantically and geometrically meaningful
transitions that harmoniously blend with the existing scene. A 2D layout
conditioning mechanism is used to control the placement and arrangement of
scene elements. Experimental results indicate that BlockFusion is capable of
generating diverse, geometrically consistent and unbounded large 3D scenes with
unprecedented high-quality shapes in both indoor and outdoor scenarios.
65b9cc5c6c920db4ba0acb47
Zhennan Wu
admin_assigned
2024-01-31T09:47:53.806Z
651c274b3edb90d4e7b0d6a9
/avatars/2fbf87db37f5a90462085553303cd23f.svg
zhennan wu
user
swwzn
65b9cc5c6c920db4ba0acb48
Yang Li
65b9cc5c6c920db4ba0acb49
Han Yan
65b9cc5c6c920db4ba0acb4a
Taizhang Shang
65b9cc5c6c920db4ba0acb4b
Weixuan Sun
admin_assigned
2024-01-31T09:48:27.402Z
640c4d2d536d9fe0f000b209
/avatars/419e59b78f7fc7f6b3ac343398d11ec1.svg
Weixuan Sun
user
weixuansun
65b9cc5c6c920db4ba0acb4c
Senbo Wang
65b9cc5c6c920db4ba0acb4d
Ruikai Cui
65b9cc5c6c920db4ba0acb4e
Weizhe Liu
65b9cc5c6c920db4ba0acb4f
Hiroyuki Sato
65b9cc5c6c920db4ba0acb50
Hongdong Li
65b9cc5c6c920db4ba0acb51
Pan Ji
claimed_verified
2024-01-31T11:48:16.048Z
6444d87e5691ca69b0d8f56a
/avatars/78d4d2b36d629a8e6ad833e102bb86f7.svg
Peter Ji
user
peterji
BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane
  Extrapolation
We present BlockFusion, a diffusion-based model that generates 3D scenes as
unit blocks and seamlessly incorporates new blocks to extend the scene.
BlockFusion is trained using datasets of 3D blocks that are randomly cropped
from complete 3D scene meshes. Through per-block fitting, all training blocks
are converted into the hybrid neural fields: with a tri-plane containing the
geometry features, followed by a Multi-layer Perceptron (MLP) for decoding the
signed distance values. A variational auto-encoder is employed to compress the
tri-planes into the latent tri-plane space, on which the denoising diffusion
process is performed. Diffusion applied to the latent representations allows
for high-quality and diverse 3D scene generation. To expand a scene during
generation, one needs only to append empty blocks to overlap with the current
scene and extrapolate existing latent tri-planes to populate new blocks. The
extrapolation is done by conditioning the generation process with the feature
samples from the overlapping tri-planes during the denoising iterations. Latent
tri-plane extrapolation produces semantically and geometrically meaningful
transitions that harmoniously blend with the existing scene. A 2D layout
conditioning mechanism is used to control the placement and arrangement of
scene elements. Experimental results indicate that BlockFusion is capable of
generating diverse, geometrically consistent and unbounded large 3D scenes with
unprecedented high-quality shapes in both indoor and outdoor scenarios.
BlockFusion, a diffusion-based model, generates and expands 3D scenes using latent tri-plane representations and a 2D layout conditioning mechanism.
diffusion-based model 3D scenes unit blocks hybrid neural fields tri-plane Multi-layer Perceptron MLP variational auto-encoder latent tri-plane space denoising diffusion latent tri-plane extrapolation 2D layout conditioning mechanism