title: Describing Differences in Image Sets with Natural Language
summary: How do two sets of images differ? Discerning set-level differences is crucial
  for understanding model behaviors and analyzing datasets, yet manually sifting
  through thousands of images is impractical. To aid in this discovery process,
  we explore the task of automatically describing the differences between two
  sets of images, which we term Set Difference Captioning. This task
  takes in image sets D_A and D_B, and outputs a description that is more
  often true on D_A than D_B. We outline a two-stage approach that first
  proposes candidate difference descriptions from image sets and then re-ranks
  the candidates by checking how well they can differentiate the two sets. We
  introduce VisDiff, which first captions the images and prompts a language model
  to propose candidate descriptions, then re-ranks these descriptions using CLIP.
  To evaluate VisDiff, we collect VisDiffBench, a dataset with 187 paired image
  sets with ground truth difference descriptions. We apply VisDiff to various
  domains, such as comparing datasets (e.g., ImageNet vs. ImageNetV2), comparing
  classification models (e.g., zero-shot CLIP vs. supervised ResNet), summarizing
  model failure modes (supervised ResNet), characterizing differences between
  generative models (e.g., StableDiffusionV1 and V2), and discovering what makes
  images memorable. Using VisDiff, we are able to find interesting and previously
  unknown differences in datasets and models, demonstrating its utility in
  revealing nuanced insights.
paper_authors._id: 656fd7dc6871b53dba94535a
paper_authors.name: Lisa Dunlap
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-12-06T02:13:36.794Z
paper_authors._id: 63f7ccb8ad35fabd915171a8
paper_authors.avatarUrl: /avatars/786bc609e6305730f0f2025ae5c27f82.svg
paper_authors.fullname: Lisa Dunlap
paper_authors.type: user
paper_authors.user: lisabdunlap
paper_authors._id: 656fd7dc6871b53dba94535b
paper_authors.name: Yuhui Zhang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-12-06T02:59:27.624Z
paper_authors._id: 62da55164398e21bf7f0e292
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62da55164398e21bf7f0e292/xjKkG8IA2IZZqCdjApSh3.jpeg
paper_authors.fullname: Yuhui Zhang
paper_authors.type: user
paper_authors.user: yuhuizhang
paper_authors._id: 656fd7dc6871b53dba94535c
paper_authors.name: Xiaohan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:45:02.498Z
paper_authors._id: 65703fab7f50602340d23704
paper_authors.avatarUrl: /avatars/324c45f5fba9cd8c38a89b30427c06b4.svg
paper_authors.fullname: Xiaohan Wang
paper_authors.type: user
paper_authors.user: nicholswang
paper_authors._id: 656fd7dc6871b53dba94535d
paper_authors.name: Ruiqi Zhong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T11:51:56.831Z
paper_authors._id: 5eae5de74e876668a0c37779
paper_authors.avatarUrl: /avatars/cf04ab4832c6565ed085dec12fcae188.svg
paper_authors.fullname: Ruiqi Zhong
paper_authors.type: user
paper_authors.user: ruiqi-zhong
paper_authors._id: 656fd7dc6871b53dba94535e
paper_authors.name: Trevor Darrell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T11:52:06.354Z
paper_authors._id: 64cbdf02f103036e23d1c7f3
paper_authors.avatarUrl: /avatars/496069463900dea20929b57381182d39.svg
paper_authors.fullname: Trevor Darrell
paper_authors.type: user
paper_authors.user: trevordarrell
paper_authors._id: 656fd7dc6871b53dba94535f
paper_authors.name: Jacob Steinhardt
paper_authors._id: 656fd7dc6871b53dba945360
paper_authors.name: Joseph E. Gonzalez
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-12-06T02:09:35.113Z
paper_authors._id: 645d2e8401f4eaab2a0878ce
paper_authors.avatarUrl: /avatars/1273c5fb607b4b622a746a42692fa632.svg
paper_authors.fullname: Joseph E. Gonzalez
paper_authors.type: user
paper_authors.user: ProfJoeyG
paper_authors._id: 656fd7dc6871b53dba945361
paper_authors.name: Serena Yeung-Levy
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-01-07T02:05:14.400Z
paper_authors._id: 677c8b2e92550a07fcad0f50
paper_authors.avatarUrl: /avatars/2be26e8f25e98cfe5b1d227ee0409cd0.svg
paper_authors.fullname: Serena Yeung-Levy
paper_authors.type: user
paper_authors.user: yeunglevy
paper_title: Describing Differences in Image Sets with Natural Language
paper_summary: How do two sets of images differ? Discerning set-level differences is crucial
  for understanding model behaviors and analyzing datasets, yet manually sifting
  through thousands of images is impractical. To aid in this discovery process,
  we explore the task of automatically describing the differences between two
  sets of images, which we term Set Difference Captioning. This task
  takes in image sets D_A and D_B, and outputs a description that is more
  often true on D_A than D_B. We outline a two-stage approach that first
  proposes candidate difference descriptions from image sets and then re-ranks
  the candidates by checking how well they can differentiate the two sets. We
  introduce VisDiff, which first captions the images and prompts a language model
  to propose candidate descriptions, then re-ranks these descriptions using CLIP.
  To evaluate VisDiff, we collect VisDiffBench, a dataset with 187 paired image
  sets with ground truth difference descriptions. We apply VisDiff to various
  domains, such as comparing datasets (e.g., ImageNet vs. ImageNetV2), comparing
  classification models (e.g., zero-shot CLIP vs. supervised ResNet), summarizing
  model failure modes (supervised ResNet), characterizing differences between
  generative models (e.g., StableDiffusionV1 and V2), and discovering what makes
  images memorable. Using VisDiff, we are able to find interesting and previously
  unknown differences in datasets and models, demonstrating its utility in
  revealing nuanced insights.
paper_ai_summary: VisDiff automatically generates descriptions of differences between two sets of images, aiding in the analysis of datasets and model behaviors across various applications.
paper_ai_keywords: Set Difference Captioning VisDiff candidate descriptions language model CLIP VisDiffBench zero-shot CLIP supervised ResNet StableDiffusion model failure modes generative models image memorability