DreamRenderer: Taming Multi-Instance Attribute Control in Large-Scale
  Text-to-Image Models
Image-conditioned generation methods, such as depth- and canny-conditioned
approaches, have demonstrated remarkable abilities for precise image synthesis.
However, existing models still struggle to accurately control the content of
multiple instances (or regions). Even state-of-the-art models like FLUX and
3DIS face challenges, such as attribute leakage between instances, which limits
user control. To address these issues, we introduce DreamRenderer, a
training-free approach built upon the FLUX model. DreamRenderer enables users
to control the content of each instance via bounding boxes or masks, while
ensuring overall visual harmony. We propose two key innovations: 1) Bridge
Image Tokens for Hard Text Attribute Binding, which uses replicated image
tokens as bridge tokens to ensure that T5 text embeddings, pre-trained solely
on text data, bind the correct visual attributes for each instance during Joint
Attention; 2) Hard Image Attribute Binding applied only to vital layers.
Through our analysis of FLUX, we identify the critical layers responsible for
instance attribute rendering and apply Hard Image Attribute Binding only in
these layers, using soft binding in the others. This approach ensures precise
control while preserving image quality. Evaluations on the COCO-POS and
COCO-MIG benchmarks demonstrate that DreamRenderer improves the Image Success
Ratio by 17.7% over FLUX and enhances the performance of layout-to-image models
like GLIGEN and 3DIS by up to 26.8%. Project Page:
https://limuloo.github.io/DreamRenderer/.
67d8e23afa59a8b15a9057e8
Dewei Zhou
admin_assigned
2025-03-18T09:20:39.038Z
65eaa1e2b11eeb516a973508
/avatars/beecd135bb940fdc02406f9063b3fa67.svg
Dewei Zhou
user
limuloo1999
67d8e23afa59a8b15a9057e9
Mingwei Li
admin_assigned
2025-03-18T09:20:46.810Z
64551bc2c9c0dcc8c2484cf6
/avatars/0d1ed4f4502f6f54ac6ba071e4c9a220.svg
Mingwei Li
user
aiJojosh
67d8e23afa59a8b15a9057ea
Zongxin Yang
admin_assigned
2025-03-18T09:20:52.869Z
619bf9b3cbedb87e1a92fb3b
/avatars/ee280db0232e21416c948ab9a9a2344e.svg
Zongxin Yang
user
z-x-yang
67d8e23afa59a8b15a9057eb
Yi Yang
DreamRenderer: Taming Multi-Instance Attribute Control in Large-Scale
  Text-to-Image Models
Image-conditioned generation methods, such as depth- and canny-conditioned
approaches, have demonstrated remarkable abilities for precise image synthesis.
However, existing models still struggle to accurately control the content of
multiple instances (or regions). Even state-of-the-art models like FLUX and
3DIS face challenges, such as attribute leakage between instances, which limits
user control. To address these issues, we introduce DreamRenderer, a
training-free approach built upon the FLUX model. DreamRenderer enables users
to control the content of each instance via bounding boxes or masks, while
ensuring overall visual harmony. We propose two key innovations: 1) Bridge
Image Tokens for Hard Text Attribute Binding, which uses replicated image
tokens as bridge tokens to ensure that T5 text embeddings, pre-trained solely
on text data, bind the correct visual attributes for each instance during Joint
Attention; 2) Hard Image Attribute Binding applied only to vital layers.
Through our analysis of FLUX, we identify the critical layers responsible for
instance attribute rendering and apply Hard Image Attribute Binding only in
these layers, using soft binding in the others. This approach ensures precise
control while preserving image quality. Evaluations on the COCO-POS and
COCO-MIG benchmarks demonstrate that DreamRenderer improves the Image Success
Ratio by 17.7% over FLUX and enhances the performance of layout-to-image models
like GLIGEN and 3DIS by up to 26.8%. Project Page:
https://limuloo.github.io/DreamRenderer/.
DreamRenderer enhances image-conditioned generation by precisely controlling instance attributes using bounding boxes and masks, improving the Image Success Ratio and layout-to-image performance.
depth-conditioned canny-conditioned attribute leakage DreamRenderer Bridge Image Tokens T5 text embeddings Joint Attention Hard Image Attribute Binding COCO-POS COCO-MIG Image Success Ratio GLIGEN 3DIS