MediFact at MEDIQA-M3G 2024: Medical Question Answering in Dermatology with Multimodal Learning
<|reference_start|>MediFact at MEDIQA-M3G 2024: Medical Question Answering in Dermatology with Multimodal Learning: The MEDIQA-M3G 2024 challenge necessitates novel solutions for Multilingual & Multimodal Medical Answer Generation in dermatology (wai Yim et al., 2024a). This paper addresses the limitations of traditional methods by proposing a weakly supervised learning approach for open-ended medical question-answering (QA). Our system leverages readily available MEDIQA-M3G images via a VGG16-CNN-SVM model, enabling multilingual (English, Chinese, Spanish) learning of informative skin condition representations. Using pre-trained QA models, we further bridge the gap between visual and textual information through multimodal fusion. This approach tackles complex, open-ended questions even without predefined answer choices. We empower the generation of comprehensive answers by feeding the ViT-CLIP model with multiple responses alongside images. This work advances medical QA research, paving the way for clinical decision support systems and ultimately improving healthcare delivery.<|reference_end|>
@article{saeed2024medifact,
    title={MediFact at MEDIQA-M3G 2024: Medical Question Answering in Dermatology
  with Multimodal Learning},
    author={Nadia Saeed},
    journal={arXiv preprint arXiv:2405.01583},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.01583},
    primaryClass={cs.CL cs.AI cs.CV cs.LG}
}