title: Diffusion Models as Data Mining Tools
summary: This paper demonstrates how to use generative models trained for image
  synthesis as tools for visual data mining. Our insight is that since
  contemporary generative models learn an accurate representation of their
  training data, we can use them to summarize the data by mining for visual
  patterns. Concretely, we show that after finetuning conditional diffusion
  models to synthesize images from a specific dataset, we can use these models to
  define a typicality measure on that dataset. This measure assesses how typical
  visual elements are for different data labels, such as geographic location,
  time stamps, semantic labels, or even the presence of a disease. This
  analysis-by-synthesis approach to data mining has two key advantages. First, it
  scales much better than traditional correspondence-based approaches since it
  does not require explicitly comparing all pairs of visual elements. Second,
  while most previous works on visual data mining focus on a single dataset, our
  approach works on diverse datasets in terms of content and scale, including a
  historical car dataset, a historical face dataset, a large worldwide
  street-view dataset, and an even larger scene dataset. Furthermore, our
  approach allows for translating visual elements across class labels and
  analyzing consistent changes.
paper_authors._id: 66b2d253fa7069c7f1eef4df
paper_authors.name: Ioannis Siglidis
paper_authors._id: 66b2d253fa7069c7f1eef4e0
paper_authors.name: Aleksander Holynski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T12:24:00.571Z
paper_authors._id: 62f6e2792e53c2efd33faa92
paper_authors.avatarUrl: /avatars/85d94b4022577747b8d2d10a82c2f3c7.svg
paper_authors.fullname: Aleksander Holynski
paper_authors.type: user
paper_authors.user: holynski
paper_authors._id: 66b2d253fa7069c7f1eef4e1
paper_authors.name: Alexei A. Efros
paper_authors._id: 66b2d253fa7069c7f1eef4e2
paper_authors.name: Mathieu Aubry
paper_authors._id: 66b2d253fa7069c7f1eef4e3
paper_authors.name: Shiry Ginosar
paper_title: Diffusion Models as Data Mining Tools
paper_summary: This paper demonstrates how to use generative models trained for image
  synthesis as tools for visual data mining. Our insight is that since
  contemporary generative models learn an accurate representation of their
  training data, we can use them to summarize the data by mining for visual
  patterns. Concretely, we show that after finetuning conditional diffusion
  models to synthesize images from a specific dataset, we can use these models to
  define a typicality measure on that dataset. This measure assesses how typical
  visual elements are for different data labels, such as geographic location,
  time stamps, semantic labels, or even the presence of a disease. This
  analysis-by-synthesis approach to data mining has two key advantages. First, it
  scales much better than traditional correspondence-based approaches since it
  does not require explicitly comparing all pairs of visual elements. Second,
  while most previous works on visual data mining focus on a single dataset, our
  approach works on diverse datasets in terms of content and scale, including a
  historical car dataset, a historical face dataset, a large worldwide
  street-view dataset, and an even larger scene dataset. Furthermore, our
  approach allows for translating visual elements across class labels and
  analyzing consistent changes.
paper_ai_summary: The approach uses fine-tuned conditional diffusion models to define a typicality measure for visual data elements across various labels and datasets, improving scalability and versatility in visual data mining.
paper_ai_keywords: generative models image synthesis visual data mining conditional diffusion models typicality measure geographic location time stamps semantic labels disease presence analysis-by-synthesis scalability historical car dataset historical face dataset worldwide street-view dataset scene dataset visual element translation consistent changes