title: Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD
    Generalization
summary: Existing vision-language models exhibit strong generalization on a variety of
  visual domains and tasks. However, such models mainly perform zero-shot
  recognition in a closed-set manner, and thus struggle to handle open-domain
  visual concepts by design. There are recent finetuning methods, such as prompt
  learning, that not only study the discrimination between in-distribution (ID)
  and out-of-distribution (OOD) samples, but also show some improvements in both
  ID and OOD accuracies. In this paper, we first demonstrate that vision-language
  models, after long enough finetuning but without proper regularization, tend to
  overfit the known classes in the given dataset, with degraded performance on
  unknown classes. Then we propose a novel approach OGEN to address this pitfall,
  with the main focus on improving the OOD GENeralization of finetuned models.
  Specifically, a class-conditional feature generator is introduced to synthesize
  OOD features using just the class name of any unknown class. Such synthesized
  features will provide useful knowledge about unknowns and help regularize the
  decision boundary between ID and OOD data when optimized jointly. Equally
  important is our adaptive self-distillation mechanism to regularize our feature
  generation model during joint optimization, i.e., adaptively transferring
  knowledge between model states to further prevent overfitting. Experiments
  validate that our method yields convincing gains in OOD generalization
  performance in different settings.
paper_authors._id: 65b868edcda408e65ffacfeb
paper_authors.name: Yuhang Zang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-30T10:05:53.449Z
paper_authors._id: 63859cf3b2906edaf83af9f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/iUQm5FAomzqYi6fkqIn9F.jpeg
paper_authors.fullname: Yuhang Zang
paper_authors.type: user
paper_authors.user: yuhangzang
paper_authors._id: 65b868edcda408e65ffacfec
paper_authors.name: Hanlin Goh
paper_authors._id: 65b868edcda408e65ffacfed
paper_authors.name: Josh Susskind
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-22T16:30:51.561Z
paper_authors._id: 6470d2247fd7ecdbd0ec3cc9
paper_authors.avatarUrl: /avatars/52c5eca12499a1aa9bd49c43d4f20685.svg
paper_authors.fullname: Joshua M. Susskind
paper_authors.type: user
paper_authors.user: jsusskind
paper_authors._id: 65b868edcda408e65ffacfee
paper_authors.name: Chen Huang
paper_title: Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD
    Generalization
paper_summary: Existing vision-language models exhibit strong generalization on a variety of
  visual domains and tasks. However, such models mainly perform zero-shot
  recognition in a closed-set manner, and thus struggle to handle open-domain
  visual concepts by design. There are recent finetuning methods, such as prompt
  learning, that not only study the discrimination between in-distribution (ID)
  and out-of-distribution (OOD) samples, but also show some improvements in both
  ID and OOD accuracies. In this paper, we first demonstrate that vision-language
  models, after long enough finetuning but without proper regularization, tend to
  overfit the known classes in the given dataset, with degraded performance on
  unknown classes. Then we propose a novel approach OGEN to address this pitfall,
  with the main focus on improving the OOD GENeralization of finetuned models.
  Specifically, a class-conditional feature generator is introduced to synthesize
  OOD features using just the class name of any unknown class. Such synthesized
  features will provide useful knowledge about unknowns and help regularize the
  decision boundary between ID and OOD data when optimized jointly. Equally
  important is our adaptive self-distillation mechanism to regularize our feature
  generation model during joint optimization, i.e., adaptively transferring
  knowledge between model states to further prevent overfitting. Experiments
  validate that our method yields convincing gains in OOD generalization
  performance in different settings.
paper_ai_keywords: vision-language models zero-shot recognition closed-set manner open-domain visual concepts prompt learning in-distribution (ID) out-of-distribution (OOD) finetuning overfitting class-conditional feature generator OOD GENeralization (OGEN) feature generation model self-distillation mechanism OOD generalization performance