ConceptAttention: Diffusion Transformers Learn Highly Interpretable
  Features
Do the rich representations of multi-modal diffusion transformers (DiTs)
exhibit unique properties that enhance their interpretability? We introduce
ConceptAttention, a novel method that leverages the expressive power of DiT
attention layers to generate high-quality saliency maps that precisely locate
textual concepts within images. Without requiring additional training,
ConceptAttention repurposes the parameters of DiT attention layers to produce
highly contextualized concept embeddings, contributing the major discovery that
performing linear projections in the output space of DiT attention layers
yields significantly sharper saliency maps compared to commonly used
cross-attention mechanisms. Remarkably, ConceptAttention even achieves
state-of-the-art performance on zero-shot image segmentation benchmarks,
outperforming 11 other zero-shot interpretability methods on the
ImageNet-Segmentation dataset and on a single-class subset of PascalVOC. Our
work contributes the first evidence that the representations of multi-modal DiT
models like Flux are highly transferable to vision tasks like segmentation,
even outperforming multi-modal foundation models like CLIP.
67a6431d0fdd5543151da7d2
Alec Helbling
claimed_verified
2025-02-08T13:56:13.003Z
62d757a22d32f0bff5710596
/avatars/f3e05ea4fb853420923e04b6bf3a1a6e.svg
Alec Helbling
user
helblazer811
67a6431d0fdd5543151da7d3
Tuna Han Salih Meral
claimed_verified
2025-02-08T13:56:15.868Z
64f8b03f83807928d25e766f
/avatars/68fd4ee967a1673a1d78a7581be8b3da.svg
Tuna Han Salih Meral
user
tmeral
67a6431d0fdd5543151da7d4
Ben Hoover
67a6431d0fdd5543151da7d5
Pinar Yanardag
claimed_verified
2025-06-02T07:53:52.102Z
683b9c7d4568ffd33e7334e1
/avatars/046bd5a78a24f3bc4e1f6cb6076970e0.svg
Pinar Yanardag
user
Pinguar
67a6431d0fdd5543151da7d6
Duen Horng Chau
ConceptAttention: Diffusion Transformers Learn Highly Interpretable
  Features
Do the rich representations of multi-modal diffusion transformers (DiTs)
exhibit unique properties that enhance their interpretability? We introduce
ConceptAttention, a novel method that leverages the expressive power of DiT
attention layers to generate high-quality saliency maps that precisely locate
textual concepts within images. Without requiring additional training,
ConceptAttention repurposes the parameters of DiT attention layers to produce
highly contextualized concept embeddings, contributing the major discovery that
performing linear projections in the output space of DiT attention layers
yields significantly sharper saliency maps compared to commonly used
cross-attention mechanisms. Remarkably, ConceptAttention even achieves
state-of-the-art performance on zero-shot image segmentation benchmarks,
outperforming 11 other zero-shot interpretability methods on the
ImageNet-Segmentation dataset and on a single-class subset of PascalVOC. Our
work contributes the first evidence that the representations of multi-modal DiT
models like Flux are highly transferable to vision tasks like segmentation,
even outperforming multi-modal foundation models like CLIP.
ConceptAttention, a method using DiT attention layers, generates precise saliency maps for textual concepts in images, achieving state-of-the-art performance in zero-shot image segmentation without additional training.
multi-modal diffusion transformers DiTs ConceptAttention saliency maps attention layers zero-shot image segmentation ImageNet-Segmentation PascalVOC CLIP