Multimodal Prototyping for cancer survival prediction
<|reference_start|>Multimodal Prototyping for cancer survival prediction: Multimodal survival methods combining gigapixel histology whole-slide images (WSIs) and transcriptomic profiles are particularly promising for patient prognostication and stratification. Current approaches involve tokenizing the WSIs into smaller patches (>10,000 patches) and transcriptomics into gene groups, which are then integrated using a Transformer for predicting outcomes. However, this process generates many tokens, which leads to high memory requirements for computing attention and complicates post-hoc interpretability analyses. Instead, we hypothesize that we can: (1) effectively summarize the morphological content of a WSI by condensing its constituting tokens using morphological prototypes, achieving more than 300x compression; and (2) accurately characterize cellular functions by encoding the transcriptomic profile with biological pathway prototypes, all in an unsupervised fashion. The resulting multimodal tokens are then processed by a fusion network, either with a Transformer or an optimal transport cross-alignment, which now operates with a small and fixed number of tokens without approximations. Extensive evaluation on six cancer types shows that our framework outperforms state-of-the-art methods with much less computation while unlocking new interpretability analyses.<|reference_end|>
@article{song2024multimodal,
    title={Multimodal Prototyping for cancer survival prediction},
    author={Andrew H. Song, Richard J. Chen, Guillaume Jaume, Anurag J. Vaidya,
  Alexander S. Baras, Faisal Mahmood},
    journal={arXiv preprint arXiv:2407.00224},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.00224},
    primaryClass={cs.CV stat.AP}
}