title: Augmenting CLIP with Improved Visio-Linguistic Reasoning
summary: Image-text contrastive models such as CLIP are useful for a variety of
  downstream applications including zero-shot classification, image-text
  retrieval and transfer learning. However, these contrastively trained
  vision-language models often fail on compositional visio-linguistic tasks such
  as Winoground with performance equivalent to random chance. In our paper, we
  address this issue and propose a sample-efficient light-weight method called
  SDS-CLIP to improve the compositional visio-linguistic reasoning capabilities
  of CLIP. The core idea of our method is to use differentiable image
  parameterizations to fine-tune CLIP with a distillation objective from large
  text-to-image generative models such as Stable-Diffusion which are relatively
  good at visio-linguistic reasoning tasks. On the challenging Winoground
  compositional reasoning benchmark, our method improves the absolute
  visio-linguistic performance of different CLIP models by up to 7%, while on the
  ARO dataset, our method improves the visio-linguistic performance by upto 3%.
  As a byproduct of inducing visio-linguistic reasoning into CLIP, we also find
  that the zero-shot performance improves marginally on a variety of downstream
  datasets. Our method reinforces that carefully designed distillation objectives
  from generative models can be leveraged to extend existing contrastive
  image-text models with improved visio-linguistic reasoning capabilities.
paper_authors._id: 64b7422808e2452d18d71bea
paper_authors.name: Samyadeep Basu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-19T15:07:29.078Z
paper_authors._id: 6384d891cafe4e4ac4060206
paper_authors.avatarUrl: /avatars/45eb04bf6e0ed1a4daa14070411b693f.svg
paper_authors.fullname: Samyadeep
paper_authors.type: user
paper_authors.user: samyadeepbasu
paper_authors._id: 64b7422808e2452d18d71beb
paper_authors.name: Maziar Sanjabi
paper_authors._id: 64b7422808e2452d18d71bec
paper_authors.name: Daniela Massiceti
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-19T14:39:19.084Z
paper_authors._id: 64b11a7c8d03d1a4e9ce3e65
paper_authors.avatarUrl: /avatars/c678eb73f5531d3a8e25b9ec63b3d501.svg
paper_authors.fullname: Daniela Massiceti 
paper_authors.type: user
paper_authors.user: dannimassi
paper_authors._id: 64b7422808e2452d18d71bed
paper_authors.name: Shell Xu Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-19T12:11:02.274Z
paper_authors._id: 625d3b3de1176e6c77268df6
paper_authors.avatarUrl: /avatars/73deace40ca2879c3bae1897979fc63e.svg
paper_authors.fullname: Shell Xu Hu
paper_authors.type: user
paper_authors.user: hushell
paper_authors._id: 64b7422808e2452d18d71bee
paper_authors.name: Soheil Feizi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-20T07:17:48.334Z
paper_authors._id: 64b83f32047fa3db94108a9f
paper_authors.avatarUrl: /avatars/c9afdb5c4c9dc3bf15c9edb223428f5f.svg
paper_authors.fullname: Soheil Feizi
paper_authors.type: user
paper_authors.user: sfeizi
paper_title: Augmenting CLIP with Improved Visio-Linguistic Reasoning
paper_summary: Image-text contrastive models such as CLIP are useful for a variety of
  downstream applications including zero-shot classification, image-text
  retrieval and transfer learning. However, these contrastively trained
  vision-language models often fail on compositional visio-linguistic tasks such
  as Winoground with performance equivalent to random chance. In our paper, we
  address this issue and propose a sample-efficient light-weight method called
  SDS-CLIP to improve the compositional visio-linguistic reasoning capabilities
  of CLIP. The core idea of our method is to use differentiable image
  parameterizations to fine-tune CLIP with a distillation objective from large
  text-to-image generative models such as Stable-Diffusion which are relatively
  good at visio-linguistic reasoning tasks. On the challenging Winoground
  compositional reasoning benchmark, our method improves the absolute
  visio-linguistic performance of different CLIP models by up to 7%, while on the
  ARO dataset, our method improves the visio-linguistic performance by upto 3%.
  As a byproduct of inducing visio-linguistic reasoning into CLIP, we also find
  that the zero-shot performance improves marginally on a variety of downstream
  datasets. Our method reinforces that carefully designed distillation objectives
  from generative models can be leveraged to extend existing contrastive
  image-text models with improved visio-linguistic reasoning capabilities.
paper_ai_summary: A method called SDS-CLIP improves compositional visio-linguistic reasoning in CLIP using distillation from large text-to-image generative models like Stable-Diffusion.
paper_ai_keywords: CLIP zero-shot classification image-text retrieval transfer learning compositional visio-linguistic tasks Winoground sample-efficient light-weight differentiable image parameterizations distillation objective Stable-Diffusion visio-linguistic reasoning ARO dataset zero-shot performance