  Cancer prognosis and survival outcome predictions are crucial for therapeutic
response estimation and for stratifying patients into various treatment groups.
Medical domains concerned with cancer prognosis are abundant with multiple
modalities, including pathological image data and non-image data such as
genomic information. To date, multimodal learning has shown potential to
enhance clinical prediction model performance by extracting and aggregating
information from different modalities of the same subject. This approach could
outperform single modality learning, thus improving computer-aided diagnosis
and prognosis in numerous medical applications. In this work, we propose a
cross-modality attention-based multimodal fusion pipeline designed to integrate
modality-specific knowledge for patient survival prediction in non-small cell
lung cancer (NSCLC). Instead of merely concatenating or summing up the features
from different modalities, our method gauges the importance of each modality
for feature fusion with cross-modality relationship when infusing the
multimodal features. Compared with single modality, which achieved c-index of
0.5772 and 0.5885 using solely tissue image data or RNA-seq data, respectively,
the proposed fusion approach achieved c-index 0.6587 in our experiment,
showcasing the capability of assimilating modality-specific knowledge from
varied modalities.

Image and Video Processing;Computer Vision and Pattern Recognition