Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformers
Multimodal Diffusion Transformers (MM-DiTs) have achieved remarkable progress
in text-driven visual generation. However, even state-of-the-art MM-DiT models
like FLUX struggle with achieving precise alignment between text prompts and
generated content. We identify two key issues in the attention mechanism of
MM-DiT, namely 1) the suppression of cross-modal attention due to token
imbalance between visual and textual modalities and 2) the lack of
timestep-aware attention weighting, which hinder the alignment. To address
these issues, we propose Temperature-Adjusted Cross-modal Attention
(TACA), a parameter-efficient method that dynamically rebalances multimodal
interactions through temperature scaling and timestep-dependent adjustment.
When combined with LoRA fine-tuning, TACA significantly enhances text-image
alignment on the T2I-CompBench benchmark with minimal computational overhead.
We tested TACA on state-of-the-art models like FLUX and SD3.5, demonstrating
its ability to improve image-text alignment in terms of object appearance,
attribute binding, and spatial relationships. Our findings highlight the
importance of balancing cross-modal attention in improving semantic fidelity in
text-to-image diffusion models. Our codes are publicly available at
https://github.com/Vchitect/TACA
68479b0f3ec10bdd8ab4de94
Zhengyao Lv
68479b0f3ec10bdd8ab4de95
Tianlin Pan
claimed_verified
2025-06-10T11:20:57.479Z
66221f7a608fb4e7f46ecadb
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/yMCsWyTkL4Xdxe18uwMYR.jpeg
Tianlin Pan
user
ldiex
68479b0f3ec10bdd8ab4de96
Chenyang Si
claimed_verified
2025-06-10T08:43:05.835Z
635f8ed47c05eb9f59963d3a
https://cdn-avatars.huggingface.co/v1/production/uploads/635f8ed47c05eb9f59963d3a/uQf4p9N9pSaFy87Wg9v4k.jpeg
ChenyangSi
user
ChenyangSi
68479b0f3ec10bdd8ab4de97
Zhaoxi Chen
68479b0f3ec10bdd8ab4de98
Wangmeng Zuo
68479b0f3ec10bdd8ab4de99
Ziwei Liu
68479b0f3ec10bdd8ab4de9a
Kwan-Yee K. Wong
Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformers
Multimodal Diffusion Transformers (MM-DiTs) have achieved remarkable progress
in text-driven visual generation. However, even state-of-the-art MM-DiT models
like FLUX struggle with achieving precise alignment between text prompts and
generated content. We identify two key issues in the attention mechanism of
MM-DiT, namely 1) the suppression of cross-modal attention due to token
imbalance between visual and textual modalities and 2) the lack of
timestep-aware attention weighting, which hinder the alignment. To address
these issues, we propose Temperature-Adjusted Cross-modal Attention
(TACA), a parameter-efficient method that dynamically rebalances multimodal
interactions through temperature scaling and timestep-dependent adjustment.
When combined with LoRA fine-tuning, TACA significantly enhances text-image
alignment on the T2I-CompBench benchmark with minimal computational overhead.
We tested TACA on state-of-the-art models like FLUX and SD3.5, demonstrating
its ability to improve image-text alignment in terms of object appearance,
attribute binding, and spatial relationships. Our findings highlight the
importance of balancing cross-modal attention in improving semantic fidelity in
text-to-image diffusion models. Our codes are publicly available at
https://github.com/Vchitect/TACA
Temperature-Adjusted Cross-modal Attention (TACA) enhances text-image alignment in diffusion models by dynamically rebalancing multimodal interactions through temperature scaling and timestep-dependent adjustment.
Temperature-Adjusted Cross-modal Attention TACA multimodal interactions temperature scaling timestep-dependent adjustment FLUX SD3.5 T2I-CompBench semantic fidelity text-to-image diffusion models