Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image
  Generation
We propose a diffusion-based approach for Text-to-Image (T2I) generation with
interactive 3D layout control. Layout control has been widely studied to
alleviate the shortcomings of T2I diffusion models in understanding objects'
placement and relationships from text descriptions. Nevertheless, existing
approaches for layout control are limited to 2D layouts, require the user to
provide a static layout beforehand, and fail to preserve generated images under
layout changes. This makes these approaches unsuitable for applications that
require 3D object-wise control and iterative refinements, e.g., interior design
and complex scene generation. To this end, we leverage the recent advancements
in depth-conditioned T2I models and propose a novel approach for interactive 3D
layout control. We replace the traditional 2D boxes used in layout control with
3D boxes. Furthermore, we revamp the T2I task as a multi-stage generation
process, where at each stage, the user can insert, change, and move an object
in 3D while preserving objects from earlier stages. We achieve this through our
proposed Dynamic Self-Attention (DSA) module and the consistent 3D object
translation strategy. Experiments show that our approach can generate
complicated scenes based on 3D layouts, boosting the object generation success
rate over the standard depth-conditioned T2I methods by 2x. Moreover, it
outperforms other methods in comparison in preserving objects under layout
changes. Project Page: https://abdo-eldesokey.github.io/build-a-scene/
66ce90495e180b9b9c89e479
Abdelrahman Eldesokey
claimed_verified
2024-08-28T07:50:56.882Z
6577f3bacfb2207f11e847bb
/avatars/825998cfebc47d8106f633be5ad10964.svg
Abdelrahman Eldesokey
user
abdo-eldesokey
66ce90495e180b9b9c89e47a
Peter Wonka
Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image
  Generation
We propose a diffusion-based approach for Text-to-Image (T2I) generation with
interactive 3D layout control. Layout control has been widely studied to
alleviate the shortcomings of T2I diffusion models in understanding objects'
placement and relationships from text descriptions. Nevertheless, existing
approaches for layout control are limited to 2D layouts, require the user to
provide a static layout beforehand, and fail to preserve generated images under
layout changes. This makes these approaches unsuitable for applications that
require 3D object-wise control and iterative refinements, e.g., interior design
and complex scene generation. To this end, we leverage the recent advancements
in depth-conditioned T2I models and propose a novel approach for interactive 3D
layout control. We replace the traditional 2D boxes used in layout control with
3D boxes. Furthermore, we revamp the T2I task as a multi-stage generation
process, where at each stage, the user can insert, change, and move an object
in 3D while preserving objects from earlier stages. We achieve this through our
proposed Dynamic Self-Attention (DSA) module and the consistent 3D object
translation strategy. Experiments show that our approach can generate
complicated scenes based on 3D layouts, boosting the object generation success
rate over the standard depth-conditioned T2I methods by 2x. Moreover, it
outperforms other methods in comparison in preserving objects under layout
changes. Project Page: https://abdo-eldesokey.github.io/build-a-scene/
A novel diffusion-based Text-to-Image approach with interactive 3D layout control uses 3D boxes and a Dynamic Self-Attention module to improve object generation and preservation.
diffusion-based Text-to-Image T2I 3D layout control 2D boxes multi-stage generation Dynamic Self-Attention (DSA) 3D object translation