title: Diffusion Classifiers Understand Compositionality, but Conditions Apply
summary: Understanding visual scenes is fundamental to human intelligence. While
  discriminative models have significantly advanced computer vision, they often
  struggle with compositional understanding. In contrast, recent generative
  text-to-image diffusion models excel at synthesizing complex scenes, suggesting
  inherent compositional capabilities. Building on this, zero-shot diffusion
  classifiers have been proposed to repurpose diffusion models for discriminative
  tasks. While prior work offered promising results in discriminative
  compositional scenarios, these results remain preliminary due to a small number
  of benchmarks and a relatively shallow analysis of conditions under which the
  models succeed. To address this, we present a comprehensive study of the
  discriminative capabilities of diffusion classifiers on a wide range of
  compositional tasks. Specifically, our study covers three diffusion models (SD
  1.5, 2.0, and, for the first time, 3-m) spanning 10 datasets and over 30 tasks.
  Further, we shed light on the role that target dataset domains play in
  respective performance; to isolate the domain effects, we introduce a new
  diagnostic benchmark Self-Bench comprised of images created by diffusion models
  themselves. Finally, we explore the importance of timestep weighting and
  uncover a relationship between domain gap and timestep sensitivity,
  particularly for SD3-m. To sum up, diffusion classifiers understand
  compositionality, but conditions apply! Code and dataset are available at
  https://github.com/eugene6923/Diffusion-Classifiers-Compositionality.
paper_authors._id: 68343fcab624d4711d816434
paper_authors.name: Yujin Jeong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T13:18:42.290Z
paper_authors._id: 63a14dd9b5515dccd425275e
paper_authors.avatarUrl: /avatars/791342bd41112bc97b5f8164d761d259.svg
paper_authors.fullname: Jeong
paper_authors.type: user
paper_authors.user: eugene6923
paper_authors._id: 68343fcab624d4711d816435
paper_authors.name: Arnas Uselis
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T11:59:06.709Z
paper_authors._id: 6520898f7bf8cc2dd28b7a9c
paper_authors.avatarUrl: /avatars/87a29ba95b71ee2dce18e97aa85e17a1.svg
paper_authors.fullname: Arnas Uselis
paper_authors.type: user
paper_authors.user: Gigglingface
paper_authors._id: 68343fcab624d4711d816436
paper_authors.name: Seong Joon Oh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T09:00:58.023Z
paper_authors._id: 638a50450f10aa3064f03f23
paper_authors.avatarUrl: /avatars/0c068458e42950c851758a238225c3a6.svg
paper_authors.fullname: Seong Joon Oh
paper_authors.type: user
paper_authors.user: coallaoh
paper_authors._id: 68343fcab624d4711d816437
paper_authors.name: Anna Rohrbach
paper_title: Diffusion Classifiers Understand Compositionality, but Conditions Apply
paper_summary: Understanding visual scenes is fundamental to human intelligence. While
  discriminative models have significantly advanced computer vision, they often
  struggle with compositional understanding. In contrast, recent generative
  text-to-image diffusion models excel at synthesizing complex scenes, suggesting
  inherent compositional capabilities. Building on this, zero-shot diffusion
  classifiers have been proposed to repurpose diffusion models for discriminative
  tasks. While prior work offered promising results in discriminative
  compositional scenarios, these results remain preliminary due to a small number
  of benchmarks and a relatively shallow analysis of conditions under which the
  models succeed. To address this, we present a comprehensive study of the
  discriminative capabilities of diffusion classifiers on a wide range of
  compositional tasks. Specifically, our study covers three diffusion models (SD
  1.5, 2.0, and, for the first time, 3-m) spanning 10 datasets and over 30 tasks.
  Further, we shed light on the role that target dataset domains play in
  respective performance; to isolate the domain effects, we introduce a new
  diagnostic benchmark Self-Bench comprised of images created by diffusion models
  themselves. Finally, we explore the importance of timestep weighting and
  uncover a relationship between domain gap and timestep sensitivity,
  particularly for SD3-m. To sum up, diffusion classifiers understand
  compositionality, but conditions apply! Code and dataset are available at
  https://github.com/eugene6923/Diffusion-Classifiers-Compositionality.
paper_ai_summary: A study of diffusion classifiers across multiple datasets and tasks reveals their compositional understanding, highlighting domain-specific performance effects and timestep weighting importance.
paper_ai_keywords: discriminative models generative text-to-image diffusion models zero-shot diffusion classifiers compositional tasks SD 1.5 SD 2.0 SD3-m Self-Bench diagnostic benchmark timestep weighting domain gap timestep sensitivity