  In hospitals, data are siloed to specific information systems that make the
same information available under different modalities such as the different
medical imaging exams the patient undergoes (CT scans, MRI, PET, Ultrasound,
etc.) and their associated radiology reports. This offers unique opportunities
to obtain and use at train-time those multiple views of the same information
that might not always be available at test-time.
  In this paper, we propose an innovative framework that makes the most of
available data by learning good representations of a multi-modal input that are
resilient to modality dropping at test-time, using recent advances in mutual
information maximization. By maximizing cross-modal information at train time,
we are able to outperform several state-of-the-art baselines in two different
settings, medical image classification, and segmentation. In particular, our
method is shown to have a strong impact on the inference-time performance of
weaker modalities.

Computer Vision and Pattern Recognition;Artificial Intelligence