Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering
<|reference_start|>Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering: Vision-language models, while effective in general domains and showing strong performance in diverse multi-modal applications like visual question-answering (VQA), struggle to maintain the same level of effectiveness in more specialized domains, e.g., medical. We propose a medical vision-language model that integrates large vision and language models adapted for the medical domain. This model goes through three stages of parameter-efficient training using three separate biomedical and radiology multi-modal visual and text datasets. The proposed model achieves state-of-the-art performance on the SLAKE 1.0 medical VQA (MedVQA) dataset with an overall accuracy of 87.5% and demonstrates strong performance on another MedVQA dataset, VQA-RAD, achieving an overall accuracy of 73.2%.<|reference_end|>
@article{ha2024fusion,
    title={Fusion of Domain-Adapted Vision and Language Models for Medical Visual
  Question Answering},
    author={Cuong Nhat Ha, Shima Asaadi, Sanjeev Kumar Karn, Oladimeji Farri,
  Tobias Heimann and Thomas Runkler},
    journal={arXiv preprint arXiv:2404.16192},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.16192},
    primaryClass={cs.CL cs.CV}
}