title: How Well Does GPT-4V(ision) Adapt to Distribution Shifts? A Preliminary
    Investigation
summary: In machine learning, generalization against distribution shifts -- where
  deployment conditions diverge from the training scenarios -- is crucial,
  particularly in fields like climate modeling, biomedicine, and autonomous
  driving. The emergence of foundation models, distinguished by their extensive
  pretraining and task versatility, has led to an increased interest in their
  adaptability to distribution shifts. GPT-4V(ision) acts as the most advanced
  publicly accessible multimodal foundation model, with extensive applications
  across various domains, including anomaly detection, video understanding, image
  generation, and medical diagnosis. However, its robustness against data
  distributions remains largely underexplored. Addressing this gap, this study
  rigorously evaluates GPT-4V's adaptability and generalization capabilities in
  dynamic environments, benchmarking against prominent models like CLIP and
  LLaVA. We delve into GPT-4V's zero-shot generalization across 13 diverse
  datasets spanning natural, medical, and molecular domains. We further
  investigate its adaptability to controlled data perturbations and examine the
  efficacy of in-context learning as a tool to enhance its adaptation. Our
  findings delineate GPT-4V's capability boundaries in distribution shifts,
  shedding light on its strengths and limitations across various scenarios.
  Importantly, this investigation contributes to our understanding of how AI
  foundation models generalize to distribution shifts, offering pivotal insights
  into their adaptability and robustness. Code is publicly available at
  https://github.com/jameszhou-gl/gpt-4v-distribution-shift.
paper_authors._id: 657926832f2e4058ae0208cd
paper_authors.name: Zhongyi Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-13T14:06:07.430Z
paper_authors._id: 65799a1fc9ca8f018ef5fb91
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/X14vTF83NQA_gG4vPrVSh.jpeg
paper_authors.fullname: Zhongyi Han
paper_authors.type: user
paper_authors.user: zhyhan
paper_authors._id: 657926832f2e4058ae0208ce
paper_authors.name: Guanglin Zhou
paper_authors._id: 657926832f2e4058ae0208cf
paper_authors.name: Rundong He
paper_authors._id: 657926832f2e4058ae0208d0
paper_authors.name: Jindong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T10:43:49.787Z
paper_authors._id: 6204cc0d522e40b4a18d86e2
paper_authors.avatarUrl: /avatars/18daf2de5671e711dc745388dd60569d.svg
paper_authors.fullname: Jindong Wang
paper_authors.type: user
paper_authors.user: jindongwang
paper_authors._id: 657926832f2e4058ae0208d1
paper_authors.name: Xing Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T10:44:10.773Z
paper_authors._id: 64785ca82b3730b6b63b75a2
paper_authors.avatarUrl: /avatars/b0c4f7fb4d4c556118fa6b52d3615429.svg
paper_authors.fullname: Xing Xie
paper_authors.type: user
paper_authors.user: xingustc
paper_authors._id: 657926832f2e4058ae0208d2
paper_authors.name: Tailin Wu
paper_authors._id: 657926832f2e4058ae0208d3
paper_authors.name: Yilong Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T10:44:48.087Z
paper_authors._id: 640e7a27474aa6f89558d43f
paper_authors.avatarUrl: /avatars/be1166944eb2bd252b5fb8de3d56ec74.svg
paper_authors.fullname: Yilong Yin
paper_authors.type: user
paper_authors.user: starmana66
paper_authors._id: 657926832f2e4058ae0208d4
paper_authors.name: Salman Khan
paper_authors._id: 657926832f2e4058ae0208d5
paper_authors.name: Lina Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T10:45:07.625Z
paper_authors._id: 6503be4feda77aafd4b96f92
paper_authors.avatarUrl: /avatars/4115a866351738e64250bbfd99ad2761.svg
paper_authors.fullname: Lina Yao
paper_authors.type: user
paper_authors.user: LinaYao
paper_authors._id: 657926832f2e4058ae0208d6
paper_authors.name: Tongliang Liu
paper_authors._id: 657926832f2e4058ae0208d7
paper_authors.name: Kun Zhang
paper_title: How Well Does GPT-4V(ision) Adapt to Distribution Shifts? A Preliminary
    Investigation
paper_summary: In machine learning, generalization against distribution shifts -- where
  deployment conditions diverge from the training scenarios -- is crucial,
  particularly in fields like climate modeling, biomedicine, and autonomous
  driving. The emergence of foundation models, distinguished by their extensive
  pretraining and task versatility, has led to an increased interest in their
  adaptability to distribution shifts. GPT-4V(ision) acts as the most advanced
  publicly accessible multimodal foundation model, with extensive applications
  across various domains, including anomaly detection, video understanding, image
  generation, and medical diagnosis. However, its robustness against data
  distributions remains largely underexplored. Addressing this gap, this study
  rigorously evaluates GPT-4V's adaptability and generalization capabilities in
  dynamic environments, benchmarking against prominent models like CLIP and
  LLaVA. We delve into GPT-4V's zero-shot generalization across 13 diverse
  datasets spanning natural, medical, and molecular domains. We further
  investigate its adaptability to controlled data perturbations and examine the
  efficacy of in-context learning as a tool to enhance its adaptation. Our
  findings delineate GPT-4V's capability boundaries in distribution shifts,
  shedding light on its strengths and limitations across various scenarios.
  Importantly, this investigation contributes to our understanding of how AI
  foundation models generalize to distribution shifts, offering pivotal insights
  into their adaptability and robustness. Code is publicly available at
  https://github.com/jameszhou-gl/gpt-4v-distribution-shift.
paper_ai_keywords: GPT-4V multimodal foundation model anomaly detection video understanding image generation medical diagnosis zero-shot generalization datasets natural medical mitig mitig controlled data perturbations in-context learning CLIP LLaVA distribution shifts foundation models generalization capabilities dynamic environments