title: BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane
    Extrapolation
summary: We present BlockFusion, a diffusion-based model that generates 3D scenes as
  unit blocks and seamlessly incorporates new blocks to extend the scene.
  BlockFusion is trained using datasets of 3D blocks that are randomly cropped
  from complete 3D scene meshes. Through per-block fitting, all training blocks
  are converted into the hybrid neural fields: with a tri-plane containing the
  geometry features, followed by a Multi-layer Perceptron (MLP) for decoding the
  signed distance values. A variational auto-encoder is employed to compress the
  tri-planes into the latent tri-plane space, on which the denoising diffusion
  process is performed. Diffusion applied to the latent representations allows
  for high-quality and diverse 3D scene generation. To expand a scene during
  generation, one needs only to append empty blocks to overlap with the current
  scene and extrapolate existing latent tri-planes to populate new blocks. The
  extrapolation is done by conditioning the generation process with the feature
  samples from the overlapping tri-planes during the denoising iterations. Latent
  tri-plane extrapolation produces semantically and geometrically meaningful
  transitions that harmoniously blend with the existing scene. A 2D layout
  conditioning mechanism is used to control the placement and arrangement of
  scene elements. Experimental results indicate that BlockFusion is capable of
  generating diverse, geometrically consistent and unbounded large 3D scenes with
  unprecedented high-quality shapes in both indoor and outdoor scenarios.
paper_authors._id: 65b9cc5c6c920db4ba0acb47
paper_authors.name: Zhennan Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T09:47:53.806Z
paper_authors._id: 651c274b3edb90d4e7b0d6a9
paper_authors.avatarUrl: /avatars/2fbf87db37f5a90462085553303cd23f.svg
paper_authors.fullname: zhennan wu
paper_authors.type: user
paper_authors.user: swwzn
paper_authors._id: 65b9cc5c6c920db4ba0acb48
paper_authors.name: Yang Li
paper_authors._id: 65b9cc5c6c920db4ba0acb49
paper_authors.name: Han Yan
paper_authors._id: 65b9cc5c6c920db4ba0acb4a
paper_authors.name: Taizhang Shang
paper_authors._id: 65b9cc5c6c920db4ba0acb4b
paper_authors.name: Weixuan Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T09:48:27.402Z
paper_authors._id: 640c4d2d536d9fe0f000b209
paper_authors.avatarUrl: /avatars/419e59b78f7fc7f6b3ac343398d11ec1.svg
paper_authors.fullname: Weixuan Sun
paper_authors.type: user
paper_authors.user: weixuansun
paper_authors._id: 65b9cc5c6c920db4ba0acb4c
paper_authors.name: Senbo Wang
paper_authors._id: 65b9cc5c6c920db4ba0acb4d
paper_authors.name: Ruikai Cui
paper_authors._id: 65b9cc5c6c920db4ba0acb4e
paper_authors.name: Weizhe Liu
paper_authors._id: 65b9cc5c6c920db4ba0acb4f
paper_authors.name: Hiroyuki Sato
paper_authors._id: 65b9cc5c6c920db4ba0acb50
paper_authors.name: Hongdong Li
paper_authors._id: 65b9cc5c6c920db4ba0acb51
paper_authors.name: Pan Ji
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-31T11:48:16.048Z
paper_authors._id: 6444d87e5691ca69b0d8f56a
paper_authors.avatarUrl: /avatars/78d4d2b36d629a8e6ad833e102bb86f7.svg
paper_authors.fullname: Peter Ji
paper_authors.type: user
paper_authors.user: peterji
paper_title: BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane
    Extrapolation
paper_summary: We present BlockFusion, a diffusion-based model that generates 3D scenes as
  unit blocks and seamlessly incorporates new blocks to extend the scene.
  BlockFusion is trained using datasets of 3D blocks that are randomly cropped
  from complete 3D scene meshes. Through per-block fitting, all training blocks
  are converted into the hybrid neural fields: with a tri-plane containing the
  geometry features, followed by a Multi-layer Perceptron (MLP) for decoding the
  signed distance values. A variational auto-encoder is employed to compress the
  tri-planes into the latent tri-plane space, on which the denoising diffusion
  process is performed. Diffusion applied to the latent representations allows
  for high-quality and diverse 3D scene generation. To expand a scene during
  generation, one needs only to append empty blocks to overlap with the current
  scene and extrapolate existing latent tri-planes to populate new blocks. The
  extrapolation is done by conditioning the generation process with the feature
  samples from the overlapping tri-planes during the denoising iterations. Latent
  tri-plane extrapolation produces semantically and geometrically meaningful
  transitions that harmoniously blend with the existing scene. A 2D layout
  conditioning mechanism is used to control the placement and arrangement of
  scene elements. Experimental results indicate that BlockFusion is capable of
  generating diverse, geometrically consistent and unbounded large 3D scenes with
  unprecedented high-quality shapes in both indoor and outdoor scenarios.
paper_ai_summary: BlockFusion, a diffusion-based model, generates and expands 3D scenes using latent tri-plane representations and a 2D layout conditioning mechanism.
paper_ai_keywords: diffusion-based model 3D scenes unit blocks hybrid neural fields tri-plane Multi-layer Perceptron MLP variational auto-encoder latent tri-plane space denoising diffusion latent tri-plane extrapolation 2D layout conditioning mechanism