MMIST-ccRCC: A Real World Medical Dataset for the Development of Multi-Modal Systems
<|reference_start|>MMIST-ccRCC: A Real World Medical Dataset for the Development of Multi-Modal Systems: The acquisition of different data modalities can enhance our knowledge and understanding of various diseases, paving the way for a more personalized healthcare. Thus, medicine is progressively moving towards the generation of massive amounts of multi-modal data (\emph{e.g,} molecular, radiology, and histopathology). While this may seem like an ideal environment to capitalize data-centric machine learning approaches, most methods still focus on exploring a single or a pair of modalities due to a variety of reasons: i) lack of ready to use curated datasets; ii) difficulty in identifying the best multi-modal fusion strategy; and iii) missing modalities across patients. In this paper we introduce a real world multi-modal dataset called MMIST-CCRCC that comprises 2 radiology modalities (CT and MRI), histopathology, genomics, and clinical data from 618 patients with clear cell renal cell carcinoma (ccRCC). We provide single and multi-modal (early and late fusion) benchmarks in the task of 12-month survival prediction in the challenging scenario of one or more missing modalities for each patient, with missing rates that range from 26$\%$ for genomics data to more than 90$\%$ for MRI. We show that even with such severe missing rates the fusion of modalities leads to improvements in the survival forecasting. Additionally, incorporating a strategy to generate the latent representations of the missing modalities given the available ones further improves the performance, highlighting a potential complementarity across modalities. Our dataset and code are available here: https://multi-modal-ist.github.io/datasets/ccRCC<|reference_end|>
@article{mota2024mmist-ccrcc:,
    title={MMIST-ccRCC: A Real World Medical Dataset for the Development of
  Multi-Modal Systems},
    author={Tiago Mota and M. Rita Verdelho and Alceu Bissoto and Carlos Santiago
  and Catarina Barata},
    journal={arXiv preprint arXiv:2405.01658},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.01658},
    primaryClass={eess.IV cs.CV}
}