VMix: Improving Text-to-Image Diffusion Model with Cross-Attention
  Mixing Control
While diffusion models show extraordinary talents in text-to-image
generation, they may still fail to generate highly aesthetic images. More
specifically, there is still a gap between the generated images and the
real-world aesthetic images in finer-grained dimensions including color,
lighting, composition, etc. In this paper, we propose Cross-Attention Value
Mixing Control (VMix) Adapter, a plug-and-play aesthetics adapter, to upgrade
the quality of generated images while maintaining generality across visual
concepts by (1) disentangling the input text prompt into the content
description and aesthetic description by the initialization of aesthetic
embedding, and (2) integrating aesthetic conditions into the denoising process
through value-mixed cross-attention, with the network connected by
zero-initialized linear layers. Our key insight is to enhance the aesthetic
presentation of existing diffusion models by designing a superior condition
control method, all while preserving the image-text alignment. Through our
meticulous design, VMix is flexible enough to be applied to community models
for better visual performance without retraining. To validate the effectiveness
of our method, we conducted extensive experiments, showing that VMix
outperforms other state-of-the-art methods and is compatible with other
community modules (e.g., LoRA, ControlNet, and IPAdapter) for image generation.
The project page is https://vmix-diffusion.github.io/VMix/.
677365f3ed53dd3b007d7a97
Shaojin Wu
admin_assigned
2025-01-02T19:45:00.373Z
660114b38ae190912a61be5d
/avatars/abc4ab10d6f9769d2b5e697ccbf3fb70.svg
ShaojinWu
user
fenfan
677365f3ed53dd3b007d7a98
Fei Ding
677365f3ed53dd3b007d7a99
Mengqi Huang
claimed_verified
2025-03-20T10:54:50.427Z
630636bcd37ce67e0e4d1d42
/avatars/c3ba695d339eaaf9e810bfa0d9a7689a.svg
Mengqi Huang
user
CoreloneH
677365f3ed53dd3b007d7a9a
Wei Liu
claimed_verified
2025-03-27T09:05:33.863Z
64a50401bbc03d2b098648e6
/avatars/82a734c0da0f6245c9ee1ef9eba881fd.svg
liuwei.jikun
user
jikun63
677365f3ed53dd3b007d7a9b
Qian He
VMix: Improving Text-to-Image Diffusion Model with Cross-Attention
  Mixing Control
While diffusion models show extraordinary talents in text-to-image
generation, they may still fail to generate highly aesthetic images. More
specifically, there is still a gap between the generated images and the
real-world aesthetic images in finer-grained dimensions including color,
lighting, composition, etc. In this paper, we propose Cross-Attention Value
Mixing Control (VMix) Adapter, a plug-and-play aesthetics adapter, to upgrade
the quality of generated images while maintaining generality across visual
concepts by (1) disentangling the input text prompt into the content
description and aesthetic description by the initialization of aesthetic
embedding, and (2) integrating aesthetic conditions into the denoising process
through value-mixed cross-attention, with the network connected by
zero-initialized linear layers. Our key insight is to enhance the aesthetic
presentation of existing diffusion models by designing a superior condition
control method, all while preserving the image-text alignment. Through our
meticulous design, VMix is flexible enough to be applied to community models
for better visual performance without retraining. To validate the effectiveness
of our method, we conducted extensive experiments, showing that VMix
outperforms other state-of-the-art methods and is compatible with other
community modules (e.g., LoRA, ControlNet, and IPAdapter) for image generation.
The project page is https://vmix-diffusion.github.io/VMix/.
A Cross-Attention Value Mixing Control (VMix) adapter enhances the aesthetic quality of images generated by diffusion models without retraining.
diffusion models text-to-image generation aesthetic embedding value-mixed cross-attention zero-initialized linear layers community models LoRA ControlNet IPAdapter