title: Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image
    Generation
summary: We propose a diffusion-based approach for Text-to-Image (T2I) generation with
  interactive 3D layout control. Layout control has been widely studied to
  alleviate the shortcomings of T2I diffusion models in understanding objects'
  placement and relationships from text descriptions. Nevertheless, existing
  approaches for layout control are limited to 2D layouts, require the user to
  provide a static layout beforehand, and fail to preserve generated images under
  layout changes. This makes these approaches unsuitable for applications that
  require 3D object-wise control and iterative refinements, e.g., interior design
  and complex scene generation. To this end, we leverage the recent advancements
  in depth-conditioned T2I models and propose a novel approach for interactive 3D
  layout control. We replace the traditional 2D boxes used in layout control with
  3D boxes. Furthermore, we revamp the T2I task as a multi-stage generation
  process, where at each stage, the user can insert, change, and move an object
  in 3D while preserving objects from earlier stages. We achieve this through our
  proposed Dynamic Self-Attention (DSA) module and the consistent 3D object
  translation strategy. Experiments show that our approach can generate
  complicated scenes based on 3D layouts, boosting the object generation success
  rate over the standard depth-conditioned T2I methods by 2x. Moreover, it
  outperforms other methods in comparison in preserving objects under layout
  changes. Project Page: https://abdo-eldesokey.github.io/build-a-scene/
paper_authors._id: 66ce90495e180b9b9c89e479
paper_authors.name: Abdelrahman Eldesokey
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-28T07:50:56.882Z
paper_authors._id: 6577f3bacfb2207f11e847bb
paper_authors.avatarUrl: /avatars/825998cfebc47d8106f633be5ad10964.svg
paper_authors.fullname: Abdelrahman Eldesokey
paper_authors.type: user
paper_authors.user: abdo-eldesokey
paper_authors._id: 66ce90495e180b9b9c89e47a
paper_authors.name: Peter Wonka
paper_title: Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image
    Generation
paper_summary: We propose a diffusion-based approach for Text-to-Image (T2I) generation with
  interactive 3D layout control. Layout control has been widely studied to
  alleviate the shortcomings of T2I diffusion models in understanding objects'
  placement and relationships from text descriptions. Nevertheless, existing
  approaches for layout control are limited to 2D layouts, require the user to
  provide a static layout beforehand, and fail to preserve generated images under
  layout changes. This makes these approaches unsuitable for applications that
  require 3D object-wise control and iterative refinements, e.g., interior design
  and complex scene generation. To this end, we leverage the recent advancements
  in depth-conditioned T2I models and propose a novel approach for interactive 3D
  layout control. We replace the traditional 2D boxes used in layout control with
  3D boxes. Furthermore, we revamp the T2I task as a multi-stage generation
  process, where at each stage, the user can insert, change, and move an object
  in 3D while preserving objects from earlier stages. We achieve this through our
  proposed Dynamic Self-Attention (DSA) module and the consistent 3D object
  translation strategy. Experiments show that our approach can generate
  complicated scenes based on 3D layouts, boosting the object generation success
  rate over the standard depth-conditioned T2I methods by 2x. Moreover, it
  outperforms other methods in comparison in preserving objects under layout
  changes. Project Page: https://abdo-eldesokey.github.io/build-a-scene/
paper_ai_keywords: diffusion-based approach Text-to-Image (T2I) generation layout control interactive 3D layout control depth-conditioned T2I models 3D boxes multi-stage generation process Dynamic Self-Attention (DSA) module consistent 3D object translation strategy object generation success rate