Discriminative Diffusion Models as Few-shot Vision and Language Learners
Diffusion models, such as Stable Diffusion, have shown incredible performance
on text-to-image generation. Since text-to-image generation often requires
models to generate visual concepts with fine-grained details and attributes
specified in text prompts, can we leverage the powerful representations learned
by pre-trained diffusion models for discriminative tasks such as image-text
matching? To answer this question, we propose a novel approach, Discriminative
Stable Diffusion (DSD), which turns pre-trained text-to-image diffusion models
into few-shot discriminative learners. Our approach uses the cross-attention
score of a Stable Diffusion model to capture the mutual influence between
visual and textual information and fine-tune the model via attention-based
prompt learning to perform image-text matching. By comparing DSD with
state-of-the-art methods on several benchmark datasets, we demonstrate the
potential of using pre-trained diffusion models for discriminative tasks with
superior results on few-shot image-text matching.
6466d70df3e78d1d6be0576f
Xuehai He
admin_assigned
2023-05-26T09:10:20.802Z
6310493158d83e8f64dc8c55
/avatars/5f91ac4dfec0d6a5bf7bad6094f0fd0f.svg
Xuehai He
user
Xuehai
6466d70df3e78d1d6be05770
Weixi Feng
admin_assigned
2023-05-26T09:10:47.918Z
6259e9489326c60347dd2b6c
/avatars/6aae6282ddee96b899b7384154e9b0a0.svg
Weixi Feng
user
weixifeng
6466d70df3e78d1d6be05771
Tsu-Jui Fu
admin_assigned
2023-05-26T09:11:24.371Z
638cb2efc8912be69c149a46
https://cdn-avatars.huggingface.co/v1/production/uploads/638cb2efc8912be69c149a46/qxMNOWoSUG525DgNPpHcK.jpeg
Tsu-Jui Fu
user
tsujuifu
6466d70df3e78d1d6be05772
Varun Jampani
admin_assigned
2023-05-26T09:11:39.878Z
630e7cfe3fc17ffc50f6e835
/avatars/0b742ff094a09f9374fafcd97ab9e002.svg
Varun Jampani
user
varunjampani
6466d70df3e78d1d6be05773
Arjun Akula
6466d70df3e78d1d6be05774
Pradyumna Narayana
6466d70df3e78d1d6be05775
Sugato Basu
6466d70df3e78d1d6be05776
William Yang Wang
6466d70df3e78d1d6be05777
Xin Eric Wang
claimed_verified
2023-05-19T19:04:30.093Z
64679a226192d39142245e5e
/avatars/05abee0b6317f100923936ca2099e9eb.svg
Xin Eric Wang
user
xw-eric
Discriminative Diffusion Models as Few-shot Vision and Language Learners
Diffusion models, such as Stable Diffusion, have shown incredible performance
on text-to-image generation. Since text-to-image generation often requires
models to generate visual concepts with fine-grained details and attributes
specified in text prompts, can we leverage the powerful representations learned
by pre-trained diffusion models for discriminative tasks such as image-text
matching? To answer this question, we propose a novel approach, Discriminative
Stable Diffusion (DSD), which turns pre-trained text-to-image diffusion models
into few-shot discriminative learners. Our approach uses the cross-attention
score of a Stable Diffusion model to capture the mutual influence between
visual and textual information and fine-tune the model via attention-based
prompt learning to perform image-text matching. By comparing DSD with
state-of-the-art methods on several benchmark datasets, we demonstrate the
potential of using pre-trained diffusion models for discriminative tasks with
superior results on few-shot image-text matching.
A novel method, Discriminative Stable Diffusion (DSD), leverages pre-trained text-to-image diffusion models with attention-based prompt learning to achieve superior performance in few-shot image-text matching tasks.
diffusion models text-to-image generation pre-trained diffusion models Discriminative Stable Diffusion (DSD) cross-attention score attention-based prompt learning image-text matching few-shot image-text matching