Augmenting CLIP with Improved Visio-Linguistic Reasoning
Image-text contrastive models such as CLIP are useful for a variety of
downstream applications including zero-shot classification, image-text
retrieval and transfer learning. However, these contrastively trained
vision-language models often fail on compositional visio-linguistic tasks such
as Winoground with performance equivalent to random chance. In our paper, we
address this issue and propose a sample-efficient light-weight method called
SDS-CLIP to improve the compositional visio-linguistic reasoning capabilities
of CLIP. The core idea of our method is to use differentiable image
parameterizations to fine-tune CLIP with a distillation objective from large
text-to-image generative models such as Stable-Diffusion which are relatively
good at visio-linguistic reasoning tasks. On the challenging Winoground
compositional reasoning benchmark, our method improves the absolute
visio-linguistic performance of different CLIP models by up to 7%, while on the
ARO dataset, our method improves the visio-linguistic performance by upto 3%.
As a byproduct of inducing visio-linguistic reasoning into CLIP, we also find
that the zero-shot performance improves marginally on a variety of downstream
datasets. Our method reinforces that carefully designed distillation objectives
from generative models can be leveraged to extend existing contrastive
image-text models with improved visio-linguistic reasoning capabilities.
64b7422808e2452d18d71bea
Samyadeep Basu
claimed_verified
2023-07-19T15:07:29.078Z
6384d891cafe4e4ac4060206
/avatars/45eb04bf6e0ed1a4daa14070411b693f.svg
Samyadeep
user
samyadeepbasu
64b7422808e2452d18d71beb
Maziar Sanjabi
64b7422808e2452d18d71bec
Daniela Massiceti
claimed_verified
2023-07-19T14:39:19.084Z
64b11a7c8d03d1a4e9ce3e65
/avatars/c678eb73f5531d3a8e25b9ec63b3d501.svg
Daniela Massiceti 
user
dannimassi
64b7422808e2452d18d71bed
Shell Xu Hu
admin_assigned
2023-07-19T12:11:02.274Z
625d3b3de1176e6c77268df6
/avatars/73deace40ca2879c3bae1897979fc63e.svg
Shell Xu Hu
user
hushell
64b7422808e2452d18d71bee
Soheil Feizi
claimed_verified
2023-07-20T07:17:48.334Z
64b83f32047fa3db94108a9f
/avatars/c9afdb5c4c9dc3bf15c9edb223428f5f.svg
Soheil Feizi
user
sfeizi
Augmenting CLIP with Improved Visio-Linguistic Reasoning
Image-text contrastive models such as CLIP are useful for a variety of
downstream applications including zero-shot classification, image-text
retrieval and transfer learning. However, these contrastively trained
vision-language models often fail on compositional visio-linguistic tasks such
as Winoground with performance equivalent to random chance. In our paper, we
address this issue and propose a sample-efficient light-weight method called
SDS-CLIP to improve the compositional visio-linguistic reasoning capabilities
of CLIP. The core idea of our method is to use differentiable image
parameterizations to fine-tune CLIP with a distillation objective from large
text-to-image generative models such as Stable-Diffusion which are relatively
good at visio-linguistic reasoning tasks. On the challenging Winoground
compositional reasoning benchmark, our method improves the absolute
visio-linguistic performance of different CLIP models by up to 7%, while on the
ARO dataset, our method improves the visio-linguistic performance by upto 3%.
As a byproduct of inducing visio-linguistic reasoning into CLIP, we also find
that the zero-shot performance improves marginally on a variety of downstream
datasets. Our method reinforces that carefully designed distillation objectives
from generative models can be leveraged to extend existing contrastive
image-text models with improved visio-linguistic reasoning capabilities.
A method called SDS-CLIP improves compositional visio-linguistic reasoning in CLIP using distillation from large text-to-image generative models like Stable-Diffusion.
CLIP zero-shot classification image-text retrieval transfer learning compositional visio-linguistic tasks Winoground sample-efficient light-weight differentiable image parameterizations distillation objective Stable-Diffusion visio-linguistic reasoning ARO dataset zero-shot performance