title: Discriminative Diffusion Models as Few-shot Vision and Language Learners
summary: Diffusion models, such as Stable Diffusion, have shown incredible performance
  on text-to-image generation. Since text-to-image generation often requires
  models to generate visual concepts with fine-grained details and attributes
  specified in text prompts, can we leverage the powerful representations learned
  by pre-trained diffusion models for discriminative tasks such as image-text
  matching? To answer this question, we propose a novel approach, Discriminative
  Stable Diffusion (DSD), which turns pre-trained text-to-image diffusion models
  into few-shot discriminative learners. Our approach uses the cross-attention
  score of a Stable Diffusion model to capture the mutual influence between
  visual and textual information and fine-tune the model via attention-based
  prompt learning to perform image-text matching. By comparing DSD with
  state-of-the-art methods on several benchmark datasets, we demonstrate the
  potential of using pre-trained diffusion models for discriminative tasks with
  superior results on few-shot image-text matching.
paper_authors._id: 6466d70df3e78d1d6be0576f
paper_authors.name: Xuehai He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T09:10:20.802Z
paper_authors._id: 6310493158d83e8f64dc8c55
paper_authors.avatarUrl: /avatars/5f91ac4dfec0d6a5bf7bad6094f0fd0f.svg
paper_authors.fullname: Xuehai He
paper_authors.type: user
paper_authors.user: Xuehai
paper_authors._id: 6466d70df3e78d1d6be05770
paper_authors.name: Weixi Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T09:10:47.918Z
paper_authors._id: 6259e9489326c60347dd2b6c
paper_authors.avatarUrl: /avatars/6aae6282ddee96b899b7384154e9b0a0.svg
paper_authors.fullname: Weixi Feng
paper_authors.type: user
paper_authors.user: weixifeng
paper_authors._id: 6466d70df3e78d1d6be05771
paper_authors.name: Tsu-Jui Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T09:11:24.371Z
paper_authors._id: 638cb2efc8912be69c149a46
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638cb2efc8912be69c149a46/qxMNOWoSUG525DgNPpHcK.jpeg
paper_authors.fullname: Tsu-Jui Fu
paper_authors.type: user
paper_authors.user: tsujuifu
paper_authors._id: 6466d70df3e78d1d6be05772
paper_authors.name: Varun Jampani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T09:11:39.878Z
paper_authors._id: 630e7cfe3fc17ffc50f6e835
paper_authors.avatarUrl: /avatars/0b742ff094a09f9374fafcd97ab9e002.svg
paper_authors.fullname: Varun Jampani
paper_authors.type: user
paper_authors.user: varunjampani
paper_authors._id: 6466d70df3e78d1d6be05773
paper_authors.name: Arjun Akula
paper_authors._id: 6466d70df3e78d1d6be05774
paper_authors.name: Pradyumna Narayana
paper_authors._id: 6466d70df3e78d1d6be05775
paper_authors.name: Sugato Basu
paper_authors._id: 6466d70df3e78d1d6be05776
paper_authors.name: William Yang Wang
paper_authors._id: 6466d70df3e78d1d6be05777
paper_authors.name: Xin Eric Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T19:04:30.093Z
paper_authors._id: 64679a226192d39142245e5e
paper_authors.avatarUrl: /avatars/05abee0b6317f100923936ca2099e9eb.svg
paper_authors.fullname: Xin Eric Wang
paper_authors.type: user
paper_authors.user: xw-eric
paper_title: Discriminative Diffusion Models as Few-shot Vision and Language Learners
paper_summary: Diffusion models, such as Stable Diffusion, have shown incredible performance
  on text-to-image generation. Since text-to-image generation often requires
  models to generate visual concepts with fine-grained details and attributes
  specified in text prompts, can we leverage the powerful representations learned
  by pre-trained diffusion models for discriminative tasks such as image-text
  matching? To answer this question, we propose a novel approach, Discriminative
  Stable Diffusion (DSD), which turns pre-trained text-to-image diffusion models
  into few-shot discriminative learners. Our approach uses the cross-attention
  score of a Stable Diffusion model to capture the mutual influence between
  visual and textual information and fine-tune the model via attention-based
  prompt learning to perform image-text matching. By comparing DSD with
  state-of-the-art methods on several benchmark datasets, we demonstrate the
  potential of using pre-trained diffusion models for discriminative tasks with
  superior results on few-shot image-text matching.
paper_ai_summary: A novel method, Discriminative Stable Diffusion (DSD), leverages pre-trained text-to-image diffusion models with attention-based prompt learning to achieve superior performance in few-shot image-text matching tasks.
paper_ai_keywords: diffusion models text-to-image generation pre-trained diffusion models Discriminative Stable Diffusion (DSD) cross-attention score attention-based prompt learning image-text matching few-shot image-text matching