IterComp: Iterative Composition-Aware Feedback Learning from Model
  Gallery for Text-to-Image Generation
Advanced diffusion models like RPG, Stable Diffusion 3 and FLUX have made
notable strides in compositional text-to-image generation. However, these
methods typically exhibit distinct strengths for compositional generation, with
some excelling in handling attribute binding and others in spatial
relationships. This disparity highlights the need for an approach that can
leverage the complementary strengths of various models to comprehensively
improve the composition capability. To this end, we introduce IterComp, a novel
framework that aggregates composition-aware model preferences from multiple
models and employs an iterative feedback learning approach to enhance
compositional generation. Specifically, we curate a gallery of six powerful
open-source diffusion models and evaluate their three key compositional
metrics: attribute binding, spatial relationships, and non-spatial
relationships. Based on these metrics, we develop a composition-aware model
preference dataset comprising numerous image-rank pairs to train
composition-aware reward models. Then, we propose an iterative feedback
learning method to enhance compositionality in a closed-loop manner, enabling
the progressive self-refinement of both the base diffusion model and reward
models over multiple iterations. Theoretical proof demonstrates the
effectiveness and extensive experiments show our significant superiority over
previous SOTA methods (e.g., Omost and FLUX), particularly in multi-category
object composition and complex semantic alignment. IterComp opens new research
avenues in reward feedback learning for diffusion models and compositional
generation. Code: https://github.com/YangLing0818/IterComp
67073ff0327cec6882ea2ce1
Xinchen Zhang
claimed_verified
2024-10-10T08:01:19.383Z
653e5d31ffd60206c8b64bb5
https://cdn-avatars.huggingface.co/v1/production/uploads/653e5d31ffd60206c8b64bb5/bgztraPC27L6culMlJw4s.png
Xinchen Zhang
user
comin
67073ff0327cec6882ea2ce2
Ling Yang
claimed_verified
2025-02-11T15:34:27.425Z
64fde4e252e82dd432b74ce9
https://cdn-avatars.huggingface.co/v1/production/uploads/64fde4e252e82dd432b74ce9/-CQZbBP7FsPPyawYrsi4z.jpeg
Ling Yang
user
Lingaaaaaaa
67073ff0327cec6882ea2ce3
Guohao Li
67073ff0327cec6882ea2ce4
Yaqi Cai
admin_assigned
2024-10-10T08:44:03.195Z
657fe0b1f010d76b6e709110
/avatars/9c08817484cfb5b238d6833d788fa0a3.svg
YaQi Cai
user
yaqicc
67073ff0327cec6882ea2ce5
Jiake Xie
admin_assigned
2024-10-10T08:43:34.868Z
63f46f4f21eb234ab7395456
/avatars/16f444058549b0317243784e9e1e3110.svg
jiake xie
user
kelisiya
67073ff0327cec6882ea2ce6
Yong Tang
67073ff0327cec6882ea2ce7
Yujiu Yang
admin_assigned
2024-10-10T08:43:28.498Z
64ca1fe838837b12d5e529b7
/avatars/44a3ad9e59318784ac531993b5f69f6b.svg
Yujiu Yang
user
Thu-redrobot
67073ff0327cec6882ea2ce8
Mengdi Wang
67073ff0327cec6882ea2ce9
Bin Cui
IterComp: Iterative Composition-Aware Feedback Learning from Model
  Gallery for Text-to-Image Generation
Advanced diffusion models like RPG, Stable Diffusion 3 and FLUX have made
notable strides in compositional text-to-image generation. However, these
methods typically exhibit distinct strengths for compositional generation, with
some excelling in handling attribute binding and others in spatial
relationships. This disparity highlights the need for an approach that can
leverage the complementary strengths of various models to comprehensively
improve the composition capability. To this end, we introduce IterComp, a novel
framework that aggregates composition-aware model preferences from multiple
models and employs an iterative feedback learning approach to enhance
compositional generation. Specifically, we curate a gallery of six powerful
open-source diffusion models and evaluate their three key compositional
metrics: attribute binding, spatial relationships, and non-spatial
relationships. Based on these metrics, we develop a composition-aware model
preference dataset comprising numerous image-rank pairs to train
composition-aware reward models. Then, we propose an iterative feedback
learning method to enhance compositionality in a closed-loop manner, enabling
the progressive self-refinement of both the base diffusion model and reward
models over multiple iterations. Theoretical proof demonstrates the
effectiveness and extensive experiments show our significant superiority over
previous SOTA methods (e.g., Omost and FLUX), particularly in multi-category
object composition and complex semantic alignment. IterComp opens new research
avenues in reward feedback learning for diffusion models and compositional
generation. Code: https://github.com/YangLing0818/IterComp
IterComp, a novel framework aggregating preferences from multiple diffusion models using iterative feedback learning, significantly improves compositional text-to-image generation across various metrics.
diffusion models RPG Stable Diffusion 3 FLUX compositional text-to-image generation attribute binding spatial relationships non-spatial relationships composition-aware model preferences iterative feedback learning composition-aware reward models reward feedback learning