title: MaGRITTe: Manipulative and Generative 3D Realization from Image, Topview
    and Text
summary: The generation of 3D scenes from user-specified conditions offers a promising
  avenue for alleviating the production burden in 3D applications. Previous
  studies required significant effort to realize the desired scene, owing to
  limited control conditions. We propose a method for controlling and generating
  3D scenes under multimodal conditions using partial images, layout information
  represented in the top view, and text prompts. Combining these conditions to
  generate a 3D scene involves the following significant difficulties: (1) the
  creation of large datasets, (2) reflection on the interaction of multimodal
  conditions, and (3) domain dependence of the layout conditions. We decompose
  the process of 3D scene generation into 2D image generation from the given
  conditions and 3D scene generation from 2D images. 2D image generation is
  achieved by fine-tuning a pretrained text-to-image model with a small
  artificial dataset of partial images and layouts, and 3D scene generation is
  achieved by layout-conditioned depth estimation and neural radiance fields
  (NeRF), thereby avoiding the creation of large datasets. The use of a common
  representation of spatial information using 360-degree images allows for the
  consideration of multimodal condition interactions and reduces the domain
  dependence of the layout control. The experimental results qualitatively and
  quantitatively demonstrated that the proposed method can generate 3D scenes in
  diverse domains, from indoor to outdoor, according to multimodal conditions.
paper_authors._id: 660b93319d2edd43f73f32e1
paper_authors.name: Takayuki Hara
paper_authors._id: 660b93319d2edd43f73f32e2
paper_authors.name: Tatsuya Harada
paper_title: MaGRITTe: Manipulative and Generative 3D Realization from Image, Topview
    and Text
paper_summary: The generation of 3D scenes from user-specified conditions offers a promising
  avenue for alleviating the production burden in 3D applications. Previous
  studies required significant effort to realize the desired scene, owing to
  limited control conditions. We propose a method for controlling and generating
  3D scenes under multimodal conditions using partial images, layout information
  represented in the top view, and text prompts. Combining these conditions to
  generate a 3D scene involves the following significant difficulties: (1) the
  creation of large datasets, (2) reflection on the interaction of multimodal
  conditions, and (3) domain dependence of the layout conditions. We decompose
  the process of 3D scene generation into 2D image generation from the given
  conditions and 3D scene generation from 2D images. 2D image generation is
  achieved by fine-tuning a pretrained text-to-image model with a small
  artificial dataset of partial images and layouts, and 3D scene generation is
  achieved by layout-conditioned depth estimation and neural radiance fields
  (NeRF), thereby avoiding the creation of large datasets. The use of a common
  representation of spatial information using 360-degree images allows for the
  consideration of multimodal condition interactions and reduces the domain
  dependence of the layout control. The experimental results qualitatively and
  quantitatively demonstrated that the proposed method can generate 3D scenes in
  diverse domains, from indoor to outdoor, according to multimodal conditions.
paper_ai_keywords: text-to-image model 3D scene generation 2D image generation layout-conditioned depth estimation neural radiance fields (NeRF) 360-degree images multimodal conditions spatial information