title: ConceptAttention: Diffusion Transformers Learn Highly Interpretable
    Features
summary: Do the rich representations of multi-modal diffusion transformers (DiTs)
  exhibit unique properties that enhance their interpretability? We introduce
  ConceptAttention, a novel method that leverages the expressive power of DiT
  attention layers to generate high-quality saliency maps that precisely locate
  textual concepts within images. Without requiring additional training,
  ConceptAttention repurposes the parameters of DiT attention layers to produce
  highly contextualized concept embeddings, contributing the major discovery that
  performing linear projections in the output space of DiT attention layers
  yields significantly sharper saliency maps compared to commonly used
  cross-attention mechanisms. Remarkably, ConceptAttention even achieves
  state-of-the-art performance on zero-shot image segmentation benchmarks,
  outperforming 11 other zero-shot interpretability methods on the
  ImageNet-Segmentation dataset and on a single-class subset of PascalVOC. Our
  work contributes the first evidence that the representations of multi-modal DiT
  models like Flux are highly transferable to vision tasks like segmentation,
  even outperforming multi-modal foundation models like CLIP.
paper_authors._id: 67a6431d0fdd5543151da7d2
paper_authors.name: Alec Helbling
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-08T13:56:13.003Z
paper_authors._id: 62d757a22d32f0bff5710596
paper_authors.avatarUrl: /avatars/f3e05ea4fb853420923e04b6bf3a1a6e.svg
paper_authors.fullname: Alec Helbling
paper_authors.type: user
paper_authors.user: helblazer811
paper_authors._id: 67a6431d0fdd5543151da7d3
paper_authors.name: Tuna Han Salih Meral
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-08T13:56:15.868Z
paper_authors._id: 64f8b03f83807928d25e766f
paper_authors.avatarUrl: /avatars/68fd4ee967a1673a1d78a7581be8b3da.svg
paper_authors.fullname: Tuna Han Salih Meral
paper_authors.type: user
paper_authors.user: tmeral
paper_authors._id: 67a6431d0fdd5543151da7d4
paper_authors.name: Ben Hoover
paper_authors._id: 67a6431d0fdd5543151da7d5
paper_authors.name: Pinar Yanardag
paper_authors._id: 67a6431d0fdd5543151da7d6
paper_authors.name: Duen Horng Chau
paper_title: ConceptAttention: Diffusion Transformers Learn Highly Interpretable
    Features
paper_summary: Do the rich representations of multi-modal diffusion transformers (DiTs)
  exhibit unique properties that enhance their interpretability? We introduce
  ConceptAttention, a novel method that leverages the expressive power of DiT
  attention layers to generate high-quality saliency maps that precisely locate
  textual concepts within images. Without requiring additional training,
  ConceptAttention repurposes the parameters of DiT attention layers to produce
  highly contextualized concept embeddings, contributing the major discovery that
  performing linear projections in the output space of DiT attention layers
  yields significantly sharper saliency maps compared to commonly used
  cross-attention mechanisms. Remarkably, ConceptAttention even achieves
  state-of-the-art performance on zero-shot image segmentation benchmarks,
  outperforming 11 other zero-shot interpretability methods on the
  ImageNet-Segmentation dataset and on a single-class subset of PascalVOC. Our
  work contributes the first evidence that the representations of multi-modal DiT
  models like Flux are highly transferable to vision tasks like segmentation,
  even outperforming multi-modal foundation models like CLIP.
paper_ai_keywords: multi-modal diffusion transformers (DiTs) ConceptAttention attention layers saliency maps textual concepts parameters concept embeddings linear projections output space cross-attention mechanisms zero-shot image segmentation zero-shot interpretability methods ImageNet-Segmentation dataset PascalVOC multi-modal DiT models (Flux) multi-modal foundation models (CLIP)