title: RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned
    Vision-Language Models
summary: Fine-tuned vision-language models (VLMs) often capture spurious correlations
  between image features and textual attributes, resulting in degraded zero-shot
  performance at test time. Existing approaches for addressing spurious
  correlations (i) primarily operate at the global image-level rather than
  intervening directly on fine-grained image features and (ii) are predominantly
  designed for unimodal settings. In this work, we present RaVL, which takes a
  fine-grained perspective on VLM robustness by discovering and mitigating
  spurious correlations using local image features rather than operating at the
  global image level. Given a fine-tuned VLM, RaVL first discovers spurious
  correlations by leveraging a region-level clustering approach to identify
  precise image features contributing to zero-shot classification errors. Then,
  RaVL mitigates the identified spurious correlation with a novel region-aware
  loss function that enables the VLM to focus on relevant regions and ignore
  spurious relationships during fine-tuning. We evaluate RaVL on 654 VLMs with
  various model architectures, data domains, and learned spurious correlations.
  Our results show that RaVL accurately discovers (191% improvement over the
  closest baseline) and mitigates (8.2% improvement on worst-group image
  classification accuracy) spurious correlations. Qualitative evaluations on
  general-domain and medical-domain VLMs confirm our findings.
paper_authors._id: 673234ee6c51a40ff8739742
paper_authors.name: Maya Varma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:27:59.630Z
paper_authors._id: 6180c6e5000231f499c547c7
paper_authors.avatarUrl: /avatars/0255b430be744a8da236860e2a00307e.svg
paper_authors.fullname: Maya Varma
paper_authors.type: user
paper_authors.user: mvarma
paper_authors._id: 673234ee6c51a40ff8739743
paper_authors.name: Jean-Benoit Delbrouck
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-16T19:47:03.827Z
paper_authors._id: 62716952bcef985363db8485
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62716952bcef985363db8485/zJPPo5xlwZRJdEuwYsYKp.jpeg
paper_authors.fullname: JB D.
paper_authors.type: user
paper_authors.user: IAMJB
paper_authors._id: 673234ee6c51a40ff8739744
paper_authors.name: Zhihong Chen
paper_authors._id: 673234ee6c51a40ff8739745
paper_authors.name: Akshay Chaudhari
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-11-11T16:46:41.296Z
paper_authors._id: 6236533b76c8a780323af640
paper_authors.avatarUrl: /avatars/18078ad26aa44312a4927160216e5943.svg
paper_authors.fullname: Akshay Chaudhari
paper_authors.type: user
paper_authors.user: akshaysc
paper_authors._id: 673234ee6c51a40ff8739746
paper_authors.name: Curtis Langlotz
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-11-11T16:46:41.296Z
paper_authors._id: 62867edd1d99648808888d7b
paper_authors.avatarUrl: /avatars/22d764782c5169634deec6f43ece7500.svg
paper_authors.fullname: Curt Langlotz
paper_authors.type: user
paper_authors.user: cplanglotz
paper_title: RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned
    Vision-Language Models
paper_summary: Fine-tuned vision-language models (VLMs) often capture spurious correlations
  between image features and textual attributes, resulting in degraded zero-shot
  performance at test time. Existing approaches for addressing spurious
  correlations (i) primarily operate at the global image-level rather than
  intervening directly on fine-grained image features and (ii) are predominantly
  designed for unimodal settings. In this work, we present RaVL, which takes a
  fine-grained perspective on VLM robustness by discovering and mitigating
  spurious correlations using local image features rather than operating at the
  global image level. Given a fine-tuned VLM, RaVL first discovers spurious
  correlations by leveraging a region-level clustering approach to identify
  precise image features contributing to zero-shot classification errors. Then,
  RaVL mitigates the identified spurious correlation with a novel region-aware
  loss function that enables the VLM to focus on relevant regions and ignore
  spurious relationships during fine-tuning. We evaluate RaVL on 654 VLMs with
  various model architectures, data domains, and learned spurious correlations.
  Our results show that RaVL accurately discovers (191% improvement over the
  closest baseline) and mitigates (8.2% improvement on worst-group image
  classification accuracy) spurious correlations. Qualitative evaluations on
  general-domain and medical-domain VLMs confirm our findings.
paper_ai_summary: RaVL improves VLM robustness by identifying and mitigating spurious correlations at the local image feature level during fine-tuning.
paper_ai_keywords: vision-language models VLMs spurious correlations region-level clustering region-aware loss function zero-shot classification fine-tuning