title: 3DIS-FLUX: simple and efficient multi-instance generation with DiT
    rendering
summary: The growing demand for controllable outputs in text-to-image generation has
  driven significant advancements in multi-instance generation (MIG), enabling
  users to define both instance layouts and attributes. Currently, the
  state-of-the-art methods in MIG are primarily adapter-based. However, these
  methods necessitate retraining a new adapter each time a more advanced model is
  released, resulting in significant resource consumption. A methodology named
  Depth-Driven Decoupled Instance Synthesis (3DIS) has been introduced, which
  decouples MIG into two distinct phases: 1) depth-based scene construction and
  2) detail rendering with widely pre-trained depth control models. The 3DIS
  method requires adapter training solely during the scene construction phase,
  while enabling various models to perform training-free detail rendering.
  Initially, 3DIS focused on rendering techniques utilizing U-Net architectures
  such as SD1.5, SD2, and SDXL, without exploring the potential of recent
  DiT-based models like FLUX. In this paper, we present 3DIS-FLUX, an extension
  of the 3DIS framework that integrates the FLUX model for enhanced rendering
  capabilities. Specifically, we employ the FLUX.1-Depth-dev model for depth map
  controlled image generation and introduce a detail renderer that manipulates
  the Attention Mask in FLUX's Joint Attention mechanism based on layout
  information. This approach allows for the precise rendering of fine-grained
  attributes of each instance. Our experimental results indicate that 3DIS-FLUX,
  leveraging the FLUX model, outperforms the original 3DIS method, which utilized
  SD2 and SDXL, and surpasses current state-of-the-art adapter-based methods in
  terms of both performance and image quality. Project Page:
  https://limuloo.github.io/3DIS/.
paper_authors._id: 678739a51f65db189f9e75c5
paper_authors.name: Dewei Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:49:15.222Z
paper_authors._id: 65eaa1e2b11eeb516a973508
paper_authors.avatarUrl: /avatars/beecd135bb940fdc02406f9063b3fa67.svg
paper_authors.fullname: Dewei Zhou
paper_authors.type: user
paper_authors.user: limuloo1999
paper_authors._id: 678739a51f65db189f9e75c6
paper_authors.name: Ji Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-15T08:48:22.289Z
paper_authors._id: 64e99fc07e2ec711a7138262
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64e99fc07e2ec711a7138262/FmP3F8_UXgh9K-0gwS99A.jpeg
paper_authors.fullname: 谢集
paper_authors.type: user
paper_authors.user: sanaka87
paper_authors._id: 678739a51f65db189f9e75c7
paper_authors.name: Zongxin Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:49:20.390Z
paper_authors._id: 619bf9b3cbedb87e1a92fb3b
paper_authors.avatarUrl: /avatars/ee280db0232e21416c948ab9a9a2344e.svg
paper_authors.fullname: Zongxin Yang
paper_authors.type: user
paper_authors.user: z-x-yang
paper_authors._id: 678739a51f65db189f9e75c8
paper_authors.name: Yi Yang
paper_title: 3DIS-FLUX: simple and efficient multi-instance generation with DiT
    rendering
paper_summary: The growing demand for controllable outputs in text-to-image generation has
  driven significant advancements in multi-instance generation (MIG), enabling
  users to define both instance layouts and attributes. Currently, the
  state-of-the-art methods in MIG are primarily adapter-based. However, these
  methods necessitate retraining a new adapter each time a more advanced model is
  released, resulting in significant resource consumption. A methodology named
  Depth-Driven Decoupled Instance Synthesis (3DIS) has been introduced, which
  decouples MIG into two distinct phases: 1) depth-based scene construction and
  2) detail rendering with widely pre-trained depth control models. The 3DIS
  method requires adapter training solely during the scene construction phase,
  while enabling various models to perform training-free detail rendering.
  Initially, 3DIS focused on rendering techniques utilizing U-Net architectures
  such as SD1.5, SD2, and SDXL, without exploring the potential of recent
  DiT-based models like FLUX. In this paper, we present 3DIS-FLUX, an extension
  of the 3DIS framework that integrates the FLUX model for enhanced rendering
  capabilities. Specifically, we employ the FLUX.1-Depth-dev model for depth map
  controlled image generation and introduce a detail renderer that manipulates
  the Attention Mask in FLUX's Joint Attention mechanism based on layout
  information. This approach allows for the precise rendering of fine-grained
  attributes of each instance. Our experimental results indicate that 3DIS-FLUX,
  leveraging the FLUX model, outperforms the original 3DIS method, which utilized
  SD2 and SDXL, and surpasses current state-of-the-art adapter-based methods in
  terms of both performance and image quality. Project Page:
  https://limuloo.github.io/3DIS/.
paper_ai_keywords: text-to-image generation multi-instance generation (MIG) adapter-based methods Depth-Driven Decoupled Instance Synthesis (3DIS) depth-based scene construction detail rendering Depth Control models U-Net architectures SD1.5 SD2 SDXL DiT-based models FLUX model FLUX.1-Depth-dev model Attention Mask Joint Attention mechanism image quality