title: Generating Compositional Scenes via Text-to-image RGBA Instance
    Generation
summary: Text-to-image diffusion generative models can generate high quality images at
  the cost of tedious prompt engineering. Controllability can be improved by
  introducing layout conditioning, however existing methods lack layout editing
  ability and fine-grained control over object attributes. The concept of
  multi-layer generation holds great potential to address these limitations,
  however generating image instances concurrently to scene composition limits
  control over fine-grained object attributes, relative positioning in 3D space
  and scene manipulation abilities. In this work, we propose a novel multi-stage
  generation paradigm that is designed for fine-grained control, flexibility and
  interactivity. To ensure control over instance attributes, we devise a novel
  training paradigm to adapt a diffusion model to generate isolated scene
  components as RGBA images with transparency information. To build complex
  images, we employ these pre-generated instances and introduce a multi-layer
  composite generation process that smoothly assembles components in realistic
  scenes. Our experiments show that our RGBA diffusion model is capable of
  generating diverse and high quality instances with precise control over object
  attributes. Through multi-layer composition, we demonstrate that our approach
  allows to build and manipulate images from highly complex prompts with
  fine-grained control over object appearance and location, granting a higher
  degree of control than competing methods.
paper_authors._id: 673fc0c4bf2e9c35b2f51502
paper_authors.name: Alessandro Fontanella
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-22T10:02:04.397Z
paper_authors._id: 673f2840be306dfff3d306df
paper_authors.avatarUrl: /avatars/6d8f9fcaf1413e699afd82d62d4c5291.svg
paper_authors.fullname: Alessandro Fontanella
paper_authors.type: user
paper_authors.user: a-fontanella
paper_authors._id: 673fc0c4bf2e9c35b2f51503
paper_authors.name: Petru-Daniel Tudosiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:22:23.915Z
paper_authors._id: 63d44162ff1384ce6c636dce
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674854731168-noauth.jpeg
paper_authors.fullname: Petru-Daniel Tudosiu
paper_authors.type: user
paper_authors.user: danieltudosiu
paper_authors._id: 673fc0c4bf2e9c35b2f51504
paper_authors.name: Yongxin Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:22:30.375Z
paper_authors._id: 62b6bdaadd998a8b1e3df69b
paper_authors.avatarUrl: /avatars/c845b5fdc487430eedf2519675a2b455.svg
paper_authors.fullname: Yongxin Yang
paper_authors.type: user
paper_authors.user: HalleyX
paper_authors._id: 673fc0c4bf2e9c35b2f51505
paper_authors.name: Shifeng Zhang
paper_authors._id: 673fc0c4bf2e9c35b2f51506
paper_authors.name: Sarah Parisot
paper_title: Generating Compositional Scenes via Text-to-image RGBA Instance
    Generation
paper_summary: Text-to-image diffusion generative models can generate high quality images at
  the cost of tedious prompt engineering. Controllability can be improved by
  introducing layout conditioning, however existing methods lack layout editing
  ability and fine-grained control over object attributes. The concept of
  multi-layer generation holds great potential to address these limitations,
  however generating image instances concurrently to scene composition limits
  control over fine-grained object attributes, relative positioning in 3D space
  and scene manipulation abilities. In this work, we propose a novel multi-stage
  generation paradigm that is designed for fine-grained control, flexibility and
  interactivity. To ensure control over instance attributes, we devise a novel
  training paradigm to adapt a diffusion model to generate isolated scene
  components as RGBA images with transparency information. To build complex
  images, we employ these pre-generated instances and introduce a multi-layer
  composite generation process that smoothly assembles components in realistic
  scenes. Our experiments show that our RGBA diffusion model is capable of
  generating diverse and high quality instances with precise control over object
  attributes. Through multi-layer composition, we demonstrate that our approach
  allows to build and manipulate images from highly complex prompts with
  fine-grained control over object appearance and location, granting a higher
  degree of control than competing methods.
paper_ai_keywords: text-to-image diffusion generative models prompt engineering layout conditioning multi-layer generation concurrent generation scene composition multi-stage generation paradigm RGBA images transparency information multi-layer composite generation process realistic scenes RGBA diffusion model fine-grained object attributes object appearance location complex prompts control over instance attributes isolated scene components