MoRA: LoRA Guided Multi-Modal Disease Diagnosis with Missing Modality
<|reference_start|>MoRA: LoRA Guided Multi-Modal Disease Diagnosis with Missing Modality: Multi-modal pre-trained models efficiently extract and fuse features from different modalities with low memory requirements for fine-tuning. Despite this efficiency, their application in disease diagnosis is under-explored. A significant challenge is the frequent occurrence of missing modalities, which impairs performance. Additionally, fine-tuning the entire pre-trained model demands substantial computational resources. To address these issues, we introduce Modality-aware Low-Rank Adaptation (MoRA), a computationally efficient method. MoRA projects each input to a low intrinsic dimension but uses different modality-aware up-projections for modality-specific adaptation in cases of missing modalities. Practically, MoRA integrates into the first block of the model, significantly improving performance when a modality is missing. It requires minimal computational resources, with less than 1.6% of the trainable parameters needed compared to training the entire model. Experimental results show that MoRA outperforms existing techniques in disease diagnosis, demonstrating superior performance, robustness, and training efficiency.<|reference_end|>
@article{shi2024mora:,
    title={MoRA: LoRA Guided Multi-Modal Disease Diagnosis with Missing Modality},
    author={Zhiyi Shi, Junsik Kim, Wanhua Li, Yicong Li, Hanspeter Pfister},
    journal={arXiv preprint arXiv:2408.09064},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.09064},
    primaryClass={cs.CV cs.LG}
}