title: Multimodal ChatGPT for Medical Applications: an Experimental Study of
    GPT-4V
summary: In this paper, we critically evaluate the capabilities of the
  state-of-the-art multimodal large language model, i.e., GPT-4 with Vision
  (GPT-4V), on Visual Question Answering (VQA) task. Our experiments thoroughly
  assess GPT-4V's proficiency in answering questions paired with images using
  both pathology and radiology datasets from 11 modalities (e.g. Microscopy,
  Dermoscopy, X-ray, CT, etc.) and fifteen objects of interests (brain, liver,
  lung, etc.). Our datasets encompass a comprehensive range of medical inquiries,
  including sixteen distinct question types. Throughout our evaluations, we
  devised textual prompts for GPT-4V, directing it to synergize visual and
  textual information. The experiments with accuracy score conclude that the
  current version of GPT-4V is not recommended for real-world diagnostics due to
  its unreliable and suboptimal accuracy in responding to diagnostic medical
  questions. In addition, we delineate seven unique facets of GPT-4V's behavior
  in medical VQA, highlighting its constraints within this complex arena. The
  complete details of our evaluation cases are accessible at
  https://github.com/ZhilingYan/GPT4V-Medical-Report.
paper_authors._id: 654098d4eddfff0435446689
paper_authors.name: Zhiling Yan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-31T10:51:17.165Z
paper_authors._id: 64f7e273b2c85cf9e5f231bd
paper_authors.avatarUrl: /avatars/0b04bf122b70c2b25135d45a84b1d3d6.svg
paper_authors.fullname: Zhiling Yan
paper_authors.type: user
paper_authors.user: zhy423724
paper_authors._id: 654098d4eddfff043544668a
paper_authors.name: Kai Zhang
paper_authors._id: 654098d4eddfff043544668b
paper_authors.name: Rong Zhou
paper_authors._id: 654098d4eddfff043544668c
paper_authors.name: Lifang He
paper_authors._id: 654098d4eddfff043544668d
paper_authors.name: Xiang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-31T15:05:04.157Z
paper_authors._id: 6540f68c0a4e554c1bfd4538
paper_authors.avatarUrl: /avatars/56759120016a019fe556df09559c25a8.svg
paper_authors.fullname: Xiang Li
paper_authors.type: user
paper_authors.user: xiangli-shaun
paper_authors._id: 654098d4eddfff043544668e
paper_authors.name: Lichao Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-21T08:29:28.373Z
paper_authors._id: 65a52766215aabac489e3468
paper_authors.avatarUrl: /avatars/fe05e22cd7e12e961296426434e17c76.svg
paper_authors.fullname: Lichao Sun
paper_authors.type: user
paper_authors.user: sunlichao137
paper_title: Multimodal ChatGPT for Medical Applications: an Experimental Study of
    GPT-4V
paper_summary: In this paper, we critically evaluate the capabilities of the
  state-of-the-art multimodal large language model, i.e., GPT-4 with Vision
  (GPT-4V), on Visual Question Answering (VQA) task. Our experiments thoroughly
  assess GPT-4V's proficiency in answering questions paired with images using
  both pathology and radiology datasets from 11 modalities (e.g. Microscopy,
  Dermoscopy, X-ray, CT, etc.) and fifteen objects of interests (brain, liver,
  lung, etc.). Our datasets encompass a comprehensive range of medical inquiries,
  including sixteen distinct question types. Throughout our evaluations, we
  devised textual prompts for GPT-4V, directing it to synergize visual and
  textual information. The experiments with accuracy score conclude that the
  current version of GPT-4V is not recommended for real-world diagnostics due to
  its unreliable and suboptimal accuracy in responding to diagnostic medical
  questions. In addition, we delineate seven unique facets of GPT-4V's behavior
  in medical VQA, highlighting its constraints within this complex arena. The
  complete details of our evaluation cases are accessible at
  https://github.com/ZhilingYan/GPT4V-Medical-Report.
paper_ai_keywords: multimodal large language model GPT-4 with vision GPT-4V Visual Question Answering pathology datasets radiology datasets Microscopy Dermoscopy X-ray CT medical inquiries textual prompts visual and textual information medical VQA real-world diagnostics