title: VMix: Improving Text-to-Image Diffusion Model with Cross-Attention
    Mixing Control
summary: While diffusion models show extraordinary talents in text-to-image
  generation, they may still fail to generate highly aesthetic images. More
  specifically, there is still a gap between the generated images and the
  real-world aesthetic images in finer-grained dimensions including color,
  lighting, composition, etc. In this paper, we propose Cross-Attention Value
  Mixing Control (VMix) Adapter, a plug-and-play aesthetics adapter, to upgrade
  the quality of generated images while maintaining generality across visual
  concepts by (1) disentangling the input text prompt into the content
  description and aesthetic description by the initialization of aesthetic
  embedding, and (2) integrating aesthetic conditions into the denoising process
  through value-mixed cross-attention, with the network connected by
  zero-initialized linear layers. Our key insight is to enhance the aesthetic
  presentation of existing diffusion models by designing a superior condition
  control method, all while preserving the image-text alignment. Through our
  meticulous design, VMix is flexible enough to be applied to community models
  for better visual performance without retraining. To validate the effectiveness
  of our method, we conducted extensive experiments, showing that VMix
  outperforms other state-of-the-art methods and is compatible with other
  community modules (e.g., LoRA, ControlNet, and IPAdapter) for image generation.
  The project page is https://vmix-diffusion.github.io/VMix/.
paper_authors._id: 677365f3ed53dd3b007d7a97
paper_authors.name: Shaojin Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-02T19:45:00.373Z
paper_authors._id: 660114b38ae190912a61be5d
paper_authors.avatarUrl: /avatars/abc4ab10d6f9769d2b5e697ccbf3fb70.svg
paper_authors.fullname: ShaojinWu
paper_authors.type: user
paper_authors.user: fenfan
paper_authors._id: 677365f3ed53dd3b007d7a98
paper_authors.name: Fei Ding
paper_authors._id: 677365f3ed53dd3b007d7a99
paper_authors.name: Mengqi Huang
paper_authors._id: 677365f3ed53dd3b007d7a9a
paper_authors.name: Wei Liu
paper_authors._id: 677365f3ed53dd3b007d7a9b
paper_authors.name: Qian He
paper_title: VMix: Improving Text-to-Image Diffusion Model with Cross-Attention
    Mixing Control
paper_summary: While diffusion models show extraordinary talents in text-to-image
  generation, they may still fail to generate highly aesthetic images. More
  specifically, there is still a gap between the generated images and the
  real-world aesthetic images in finer-grained dimensions including color,
  lighting, composition, etc. In this paper, we propose Cross-Attention Value
  Mixing Control (VMix) Adapter, a plug-and-play aesthetics adapter, to upgrade
  the quality of generated images while maintaining generality across visual
  concepts by (1) disentangling the input text prompt into the content
  description and aesthetic description by the initialization of aesthetic
  embedding, and (2) integrating aesthetic conditions into the denoising process
  through value-mixed cross-attention, with the network connected by
  zero-initialized linear layers. Our key insight is to enhance the aesthetic
  presentation of existing diffusion models by designing a superior condition
  control method, all while preserving the image-text alignment. Through our
  meticulous design, VMix is flexible enough to be applied to community models
  for better visual performance without retraining. To validate the effectiveness
  of our method, we conducted extensive experiments, showing that VMix
  outperforms other state-of-the-art methods and is compatible with other
  community modules (e.g., LoRA, ControlNet, and IPAdapter) for image generation.
  The project page is https://vmix-diffusion.github.io/VMix/.
paper_ai_keywords: Cross-Attention Value Mixing Control (VMix) aesthetics adapter denoising process value-mixed cross-attention aesthetic embedding zero-initialized linear layers condition control method image-text alignment LoRA ControlNet IPAdapter