  We propose and demonstrate a novel machine learning algorithm that assesses
pulmonary edema severity from chest radiographs. While large publicly available
datasets of chest radiographs and free-text radiology reports exist, only
limited numerical edema severity labels can be extracted from radiology
reports. This is a significant challenge in learning such models for image
classification. To take advantage of the rich information present in the
radiology reports, we develop a neural network model that is trained on both
images and free-text to assess pulmonary edema severity from chest radiographs
at inference time. Our experimental results suggest that the joint image-text
representation learning improves the performance of pulmonary edema assessment
compared to a supervised model trained on images only. We also show the use of
the text for explaining the image classification by the joint model. To the
best of our knowledge, our approach is the first to leverage free-text
radiology reports for improving the image model performance in this
application. Our code is available at
https://github.com/RayRuizhiLiao/joint_chestxray.

Computer Vision and Pattern Recognition