Multimodal ChatGPT for Medical Applications: an Experimental Study of
  GPT-4V
In this paper, we critically evaluate the capabilities of the
state-of-the-art multimodal large language model, i.e., GPT-4 with Vision
(GPT-4V), on Visual Question Answering (VQA) task. Our experiments thoroughly
assess GPT-4V's proficiency in answering questions paired with images using
both pathology and radiology datasets from 11 modalities (e.g. Microscopy,
Dermoscopy, X-ray, CT, etc.) and fifteen objects of interests (brain, liver,
lung, etc.). Our datasets encompass a comprehensive range of medical inquiries,
including sixteen distinct question types. Throughout our evaluations, we
devised textual prompts for GPT-4V, directing it to synergize visual and
textual information. The experiments with accuracy score conclude that the
current version of GPT-4V is not recommended for real-world diagnostics due to
its unreliable and suboptimal accuracy in responding to diagnostic medical
questions. In addition, we delineate seven unique facets of GPT-4V's behavior
in medical VQA, highlighting its constraints within this complex arena. The
complete details of our evaluation cases are accessible at
https://github.com/ZhilingYan/GPT4V-Medical-Report.
654098d4eddfff0435446689
Zhiling Yan
admin_assigned
2023-10-31T10:51:17.165Z
64f7e273b2c85cf9e5f231bd
/avatars/0b04bf122b70c2b25135d45a84b1d3d6.svg
Zhiling Yan
user
zhy423724
654098d4eddfff043544668a
Kai Zhang
654098d4eddfff043544668b
Rong Zhou
654098d4eddfff043544668c
Lifang He
654098d4eddfff043544668d
Xiang Li
claimed_verified
2023-10-31T15:05:04.157Z
6540f68c0a4e554c1bfd4538
/avatars/56759120016a019fe556df09559c25a8.svg
Xiang Li
user
xiangli-shaun
654098d4eddfff043544668e
Lichao Sun
claimed_verified
2024-03-21T08:29:28.373Z
65a52766215aabac489e3468
/avatars/fe05e22cd7e12e961296426434e17c76.svg
Lichao Sun
user
sunlichao137
Multimodal ChatGPT for Medical Applications: an Experimental Study of
  GPT-4V
In this paper, we critically evaluate the capabilities of the
state-of-the-art multimodal large language model, i.e., GPT-4 with Vision
(GPT-4V), on Visual Question Answering (VQA) task. Our experiments thoroughly
assess GPT-4V's proficiency in answering questions paired with images using
both pathology and radiology datasets from 11 modalities (e.g. Microscopy,
Dermoscopy, X-ray, CT, etc.) and fifteen objects of interests (brain, liver,
lung, etc.). Our datasets encompass a comprehensive range of medical inquiries,
including sixteen distinct question types. Throughout our evaluations, we
devised textual prompts for GPT-4V, directing it to synergize visual and
textual information. The experiments with accuracy score conclude that the
current version of GPT-4V is not recommended for real-world diagnostics due to
its unreliable and suboptimal accuracy in responding to diagnostic medical
questions. In addition, we delineate seven unique facets of GPT-4V's behavior
in medical VQA, highlighting its constraints within this complex arena. The
complete details of our evaluation cases are accessible at
https://github.com/ZhilingYan/GPT4V-Medical-Report.
GPT-4V's evaluation on medical VQA using multi-modal datasets shows unreliability and highlights several limitations.
multimodal large language model GPT-4V Visual Question Answering (VQA) pathology radiology datasets Microscopy Dermoscopy X-ray CT medical inquiries question types textual prompts visual and textual information accuracy score real-world diagnostics unique facets behavior in medical VQA