Integrated visual and text-based analysis of ophthalmology clinical cases using a large language model
Artificial intelligence (AI) has made remarkable advancements in healthcare, and in ophthalmology, where images are central to diagnosis. Deep learning algorithms have demonstrated promising performance in analyzing ocular images . For example, AI applications have been developed for the screening of strabismus , diagnosis of keratoconus and ectatic corneal disease , and cataract classification . Most past applications were trained to analyze only one type of image, mostly retinal or optic nerve-based images , , and did not integrate clinical context. Large Language Models (LLMs) such as GPT-4 by OpenAI have shown impressive abilities in free-text analysis and generation across different healthcare tasks , , and in ophthalmology specifically – . Patient history and complaints, have been established as a major component of patient diagnosis , . However, visual data such as physical examination, imaging tests and pathology, are often critical in patient evaluation . Ophthalmology in particular relies significantly on the integration of the clinical context, physical examination and imaging . Multimodal models, such as GPT-4 with vision capabilities (GPT-4 V), represent a paradigm shift by enabling simultaneous analysis of visual and textual data. The aim of our study was to evaluate the performance of GPT-4 V in diagnosing ocular conditions based on external eye photographs, both with and without supplemental clinical context.   Study design A Sheba Medical Center institutional board approval (IRB) was granted to this study (0143-23-SMC(. This was a retrospective study assessing GPT-4 multimodal (GPT-4 V, with image analysis capability) diagnostic performance on ocular images against two non-ophthalmologist physicians, without and with supplemental clinical context. The GPT-4 V model was accessed on November 3rd. Two radiology residents were each presented with the same subset of cases to provide diagnoses. Data collection and diagnostic procedure A series of 40 anonymized ocular images was curated, representing a spectrum of ocular conditions (Table ). Retinal or optic nerve pathologies were not included. The photos were obtained by a phone camera or a slit lamp microscope camera. We included only external images of the eyes. We did not include fundus photography or other ocular imaging technologies such as ocular coherence tomography (OCT), ocular ultrasound, fluorescein angiography or radiology images in this analysis. Cases were selected by a board certified ophthalmologist, to represent various pathologies and match the level of ophthalmology residents. Each participant (GPT-4 V and human physicians) separately received these images in two sequences: initially without and subsequently with added clinical context. Both GPT-4 and the physicians were asked to render a diagnosis for the images first without any clinical context and then with additional clinical information. Clinical context included age, symptoms and relevant medical history. All interactions with GPT-4 V were conducted through the OpenAI web interface. Each inquiry was initiated in a distinct instance to ensure independence of responses. The output without clinical context and the output with clinical context were tested in separate dialogue instances to ensure independence of responses. Specific prompts used were: Outcome measures The primary metric was the accuracy of diagnoses, expressed as a percentage of correct identifications. A qualitative analysis of GPT-4 V answers was also performed. All diagnostic responses from GPT-4 V and the participating physicians were evaluated for accuracy in consensus by two board-certified ophthalmologists. An additional evaluation based on class agreement analysis by anatomical region was performed. For this analysis, each case was categorized into a specific anatomical region, and the agreement with the ground truth, as determined by the board-certified ophthalmologists, was calculated both with and without clinical context. Statistical analysis Statistical computations were conducted using SPSS software for windows version 24.0 by IBM. A Fisher’s exact test was utilized to contrast the performance differences between GPT-4 V and the physicians, and to compare overall accuracy with and without context. We considered P-values less than 0.05 as indicative of statistical significance.   A Sheba Medical Center institutional board approval (IRB) was granted to this study (0143-23-SMC(. This was a retrospective study assessing GPT-4 multimodal (GPT-4 V, with image analysis capability) diagnostic performance on ocular images against two non-ophthalmologist physicians, without and with supplemental clinical context. The GPT-4 V model was accessed on November 3rd. Two radiology residents were each presented with the same subset of cases to provide diagnoses.   A series of 40 anonymized ocular images was curated, representing a spectrum of ocular conditions (Table ). Retinal or optic nerve pathologies were not included. The photos were obtained by a phone camera or a slit lamp microscope camera. We included only external images of the eyes. We did not include fundus photography or other ocular imaging technologies such as ocular coherence tomography (OCT), ocular ultrasound, fluorescein angiography or radiology images in this analysis. Cases were selected by a board certified ophthalmologist, to represent various pathologies and match the level of ophthalmology residents. Each participant (GPT-4 V and human physicians) separately received these images in two sequences: initially without and subsequently with added clinical context. Both GPT-4 and the physicians were asked to render a diagnosis for the images first without any clinical context and then with additional clinical information. Clinical context included age, symptoms and relevant medical history. All interactions with GPT-4 V were conducted through the OpenAI web interface. Each inquiry was initiated in a distinct instance to ensure independence of responses. The output without clinical context and the output with clinical context were tested in separate dialogue instances to ensure independence of responses. Specific prompts used were:   The primary metric was the accuracy of diagnoses, expressed as a percentage of correct identifications. A qualitative analysis of GPT-4 V answers was also performed. All diagnostic responses from GPT-4 V and the participating physicians were evaluated for accuracy in consensus by two board-certified ophthalmologists. An additional evaluation based on class agreement analysis by anatomical region was performed. For this analysis, each case was categorized into a specific anatomical region, and the agreement with the ground truth, as determined by the board-certified ophthalmologists, was calculated both with and without clinical context.   Statistical computations were conducted using SPSS software for windows version 24.0 by IBM. A Fisher’s exact test was utilized to contrast the performance differences between GPT-4 V and the physicians, and to compare overall accuracy with and without context. We considered P-values less than 0.05 as indicative of statistical significance.   Study cohort and pathological variance The study cohort comprised a diverse array of 40 ocular conditions presented to the AI model and non-ophthalmologist physicians for diagnosis. Mean age of patients included was 54.4 ± 23.2 years. The pathologies included are detailed in Table . Diagnostic accuracy without and with clinical context The diagnostic accuracy of GPT-4 V based on images alone was 47.5% (19/40). In comparison, Physician 1 achieved an accuracy of 60.0% (24/40) under the same conditions. Physician 2 correctly identified 57.5% (23/40) of cases (Table ). When clinical context was included, GPT-4 V’s diagnostic accuracy improved to 67.5% (27/40). Physician 1’s accuracy was 72.5% (29/40), and Physician 2’s accuracy was 67.5% (27/40). There was no statistically significant difference between GPT-4 and physicians’ diagnostic accuracy (Table ). Overall, for all study readers, adding context improved accuracy, as can be seen in Fig. ( p = 0.033). Qualitative analysis of GPT responses Cases in which GPT-4 V was initially wrong, but when provided with clinical context correctly altered the diagnoses included: nevus of Ota, dacryocystitis, Argentinean flag, herpes zoster pseudo-dendrite, thyroid eye disease, iris nevus, cornea foreign body, and ocular perforation. Context-enriched answers showed deeper diagnostic reasoning, and blending clinical history with visual findings. Example cases are detailed in Figs. and . When analyzing the results per anatomical region, regions that are readily visible in external photographs, such as the cornea and eyelids had high class agreement. In contrast, regions that can only be indirectly inferred from external images, such as the lacrimal system and orbit had relatively lower class agreement. A detailed breakdown of class agreement by anatomical region is provided in Table .   The study cohort comprised a diverse array of 40 ocular conditions presented to the AI model and non-ophthalmologist physicians for diagnosis. Mean age of patients included was 54.4 ± 23.2 years. The pathologies included are detailed in Table .   The diagnostic accuracy of GPT-4 V based on images alone was 47.5% (19/40). In comparison, Physician 1 achieved an accuracy of 60.0% (24/40) under the same conditions. Physician 2 correctly identified 57.5% (23/40) of cases (Table ). When clinical context was included, GPT-4 V’s diagnostic accuracy improved to 67.5% (27/40). Physician 1’s accuracy was 72.5% (29/40), and Physician 2’s accuracy was 67.5% (27/40). There was no statistically significant difference between GPT-4 and physicians’ diagnostic accuracy (Table ). Overall, for all study readers, adding context improved accuracy, as can be seen in Fig. ( p = 0.033).   Cases in which GPT-4 V was initially wrong, but when provided with clinical context correctly altered the diagnoses included: nevus of Ota, dacryocystitis, Argentinean flag, herpes zoster pseudo-dendrite, thyroid eye disease, iris nevus, cornea foreign body, and ocular perforation. Context-enriched answers showed deeper diagnostic reasoning, and blending clinical history with visual findings. Example cases are detailed in Figs. and . When analyzing the results per anatomical region, regions that are readily visible in external photographs, such as the cornea and eyelids had high class agreement. In contrast, regions that can only be indirectly inferred from external images, such as the lacrimal system and orbit had relatively lower class agreement. A detailed breakdown of class agreement by anatomical region is provided in Table .   This study evaluated multimodal GPT-4 for clinical diagnosis in ophthalmology based on patient ocular images and clinical context. There are several important findings: (1) GPT-4 V showed capability for ocular image analysis, correctly identifying 48.5% (16/33) of cases based on images alone. (2) When clinical context was added, the accuracy of the model improved to 69.7%. (3) GPT-4 V performance on ophthalmology cases was comparable to non-ophthalmology physicians. A multimodal algorithm that synergizes clinical text with images signals a groundbreaking advance in medical image analysis. The integration of visual and textual data imitates a human decision making process. This is relevant to all medical specialties, but is mostly prominent in specialties that rely on pattern recognition, such as ophthalmology, dermatology, radiology and pathology. These are also the specialties that are at the forefront of AI applications in healthcare, with variable algorithms available and being evaluated for medical images processing , . With an added value of textual analysis, LLMs may ultimately surpass current algorithms that analyze images only . Although there are few recent publications on multimodal GPT-4 V, to the best of our knowledge, our study is the first to evaluate actual patient cases instead of relying on images available in medical question repositories . In this study, GPT-4 V and physicians’ performance improved with clinical context. This reinforces the long-standing medical principle that clinical history is key for accurate diagnosis . Consequently, the potential of multimodal LLMs in ophthalmology is vast. With improvement, such an algorithm can be used as a decision support tool for physicians in diagnosis and management planning. It can also be used in research for cohort generation, enabling creation of large datasets that include textual data with images findings. A multimodal LLM might also significantly impact education in ophthalmology. There is a lack of medical students’ and primary care physicians’ education and training on initial management of basic ophthalmic cases . This is supported by the results of our study, with non-ophthalmology physicians achieving diagnostic accuracies ranging between 58 and 79%. An educational tool that can provide detailed explanations of ocular examination and imaging findings could be used to improve basic understanding and recognition of ophthalmic pathologies. Patient education in this field is also lacking. Many patients already seek initial information using online and unreliable ways, which might lead to harm. A potential restricted model focused on patient education, might be able to integrate patient taken external images with a short patient submitted history to provide supervised case specific patient information, and recommended initial management. Despite their tremendous potential, there are challenges with multimodal application of LLMs in ophthalmology. First, images that cannot be anonymized, such as full-face photos, pose significant privacy issues. To address this, accessing the model would require strict security protocols. Furthermore, these models can be susceptible to cyber threats, including adversarial attacks . In addition, the models can potentially perpetuate bias in healthcare based on data from images, such as skin color or gender . This study has several limitations. First, this was a retrospective analysis of cases, chosen subjectively with potential for selection bias. Second, this was a proof-of-concept study, with a small sample size. This study included only external eye photographs and excluded imaging modalities such as OCT, fundoscopy, or radiological imaging. Cases were intentionally curated by a board-certified ophthalmologist to represent a spectrum of common external eye conditions encountered at the resident level, rather than through random sampling. This approach may have introduced a selection bias and limited the representation of diseases requiring specialized imaging, potentially influencing the overall performance. Finally, due to patient privacy concerns, we have only inserted tightly cropped images focused only on the eyes, to preserve anonymity. This may have influenced the model’s diagnostic abilities. To conclude, GPT-4 V at its current stage is not yet suitable for clinical application in ophthalmology. Nonetheless, its ability to simultaneously analyze and integrate visual and textual data is promising. Multimodal large language models like GPT-4 V have significant potential to advance patient care, education and research in ophthalmology. Future studies involving diverse imaging modalities and randomized case selection are essential to provide a more comprehensive evaluation of these models, their strengths, and limitations.   