3DIS-FLUX: simple and efficient multi-instance generation with DiT
  rendering
The growing demand for controllable outputs in text-to-image generation has
driven significant advancements in multi-instance generation (MIG), enabling
users to define both instance layouts and attributes. Currently, the
state-of-the-art methods in MIG are primarily adapter-based. However, these
methods necessitate retraining a new adapter each time a more advanced model is
released, resulting in significant resource consumption. A methodology named
Depth-Driven Decoupled Instance Synthesis (3DIS) has been introduced, which
decouples MIG into two distinct phases: 1) depth-based scene construction and
2) detail rendering with widely pre-trained depth control models. The 3DIS
method requires adapter training solely during the scene construction phase,
while enabling various models to perform training-free detail rendering.
Initially, 3DIS focused on rendering techniques utilizing U-Net architectures
such as SD1.5, SD2, and SDXL, without exploring the potential of recent
DiT-based models like FLUX. In this paper, we present 3DIS-FLUX, an extension
of the 3DIS framework that integrates the FLUX model for enhanced rendering
capabilities. Specifically, we employ the FLUX.1-Depth-dev model for depth map
controlled image generation and introduce a detail renderer that manipulates
the Attention Mask in FLUX's Joint Attention mechanism based on layout
information. This approach allows for the precise rendering of fine-grained
attributes of each instance. Our experimental results indicate that 3DIS-FLUX,
leveraging the FLUX model, outperforms the original 3DIS method, which utilized
SD2 and SDXL, and surpasses current state-of-the-art adapter-based methods in
terms of both performance and image quality. Project Page:
https://limuloo.github.io/3DIS/.
678739a51f65db189f9e75c5
Dewei Zhou
admin_assigned
2025-01-15T16:49:15.222Z
65eaa1e2b11eeb516a973508
/avatars/beecd135bb940fdc02406f9063b3fa67.svg
Dewei Zhou
user
limuloo1999
678739a51f65db189f9e75c6
Ji Xie
claimed_verified
2025-01-15T08:48:22.289Z
64e99fc07e2ec711a7138262
https://cdn-avatars.huggingface.co/v1/production/uploads/64e99fc07e2ec711a7138262/FmP3F8_UXgh9K-0gwS99A.jpeg
Ji Xie
user
sanaka87
678739a51f65db189f9e75c7
Zongxin Yang
admin_assigned
2025-01-15T16:49:20.390Z
619bf9b3cbedb87e1a92fb3b
/avatars/ee280db0232e21416c948ab9a9a2344e.svg
Zongxin Yang
user
z-x-yang
678739a51f65db189f9e75c8
Yi Yang
3DIS-FLUX: simple and efficient multi-instance generation with DiT
  rendering
The growing demand for controllable outputs in text-to-image generation has
driven significant advancements in multi-instance generation (MIG), enabling
users to define both instance layouts and attributes. Currently, the
state-of-the-art methods in MIG are primarily adapter-based. However, these
methods necessitate retraining a new adapter each time a more advanced model is
released, resulting in significant resource consumption. A methodology named
Depth-Driven Decoupled Instance Synthesis (3DIS) has been introduced, which
decouples MIG into two distinct phases: 1) depth-based scene construction and
2) detail rendering with widely pre-trained depth control models. The 3DIS
method requires adapter training solely during the scene construction phase,
while enabling various models to perform training-free detail rendering.
Initially, 3DIS focused on rendering techniques utilizing U-Net architectures
such as SD1.5, SD2, and SDXL, without exploring the potential of recent
DiT-based models like FLUX. In this paper, we present 3DIS-FLUX, an extension
of the 3DIS framework that integrates the FLUX model for enhanced rendering
capabilities. Specifically, we employ the FLUX.1-Depth-dev model for depth map
controlled image generation and introduce a detail renderer that manipulates
the Attention Mask in FLUX's Joint Attention mechanism based on layout
information. This approach allows for the precise rendering of fine-grained
attributes of each instance. Our experimental results indicate that 3DIS-FLUX,
leveraging the FLUX model, outperforms the original 3DIS method, which utilized
SD2 and SDXL, and surpasses current state-of-the-art adapter-based methods in
terms of both performance and image quality. Project Page:
https://limuloo.github.io/3DIS/.
3DIS-FLUX extends the 3DIS framework with FLUX for improved text-to-image generation, achieving better performance and image quality through depth map control and attention mask manipulation.
multi-instance generation adapter-based methods Depth-Driven Decoupled Instance Synthesis (3DIS) U-Net SD1.5 SD2 SDXL FLUX FLUX.1-Depth-dev Joint Attention mechanism Attention Mask