GPT-4V Cannot Generate Radiology Reports Yet
<|reference_start|>GPT-4V Cannot Generate Radiology Reports Yet: GPT-4V's purported strong multimodal abilities raise interests in using it to automate radiology report writing, but there lacks thorough evaluations. In this work, we perform a systematic evaluation of GPT-4V in generating radiology reports on two chest X-ray report datasets: MIMIC-CXR and IU X-Ray. We attempt to directly generate reports using GPT-4V through different prompting strategies and find that it fails terribly in both lexical metrics and clinical efficacy metrics. To understand the low performance, we decompose the task into two steps: 1) the medical image reasoning step of predicting medical condition labels from images; and 2) the report synthesis step of generating reports from (groundtruth) conditions. We show that GPT-4V's performance in image reasoning is consistently low across different prompts. In fact, the distributions of model-predicted labels remain constant regardless of which groundtruth conditions are present on the image, suggesting that the model is not interpreting chest X-rays meaningfully. Even when given groundtruth conditions in report synthesis, its generated reports are less correct and less natural-sounding than a finetuned LLaMA-2. Altogether, our findings cast doubt on the viability of using GPT-4V in a radiology workflow.<|reference_end|>
@article{jiang2024gpt-4v,
    title={GPT-4V Cannot Generate Radiology Reports Yet},
    author={Yuyang Jiang, Chacha Chen, Dang Nguyen, Benjamin M. Mervak and Chenhao
  Tan},
    journal={arXiv preprint arXiv:2407.12176},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.12176},
    primaryClass={cs.CY cs.AI cs.CL}
}