RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism
Accuracy and interpretability are two dominant features of successful predictive models. Typically, a choice must be made in favor of complex black box models such as recurrent neural networks (RNN) for accuracy versus less accurate but more interpretable traditional models such as logistic regression. This tradeoff poses challenges in medicine where both accuracy and interpretability are important. We addressed this challenge by developing the REverse Time AttentIoN model (RETAIN) for application to Electronic Health Records (EHR) data. RETAIN achieves high accuracy while remaining clinically interpretable and is based on a two-level neural attention model that detects influential past visits and significant clinical variables within those visits (e.g. key diagnoses). RETAIN mimics physician practice by attending the EHR data in a reverse time order so that recent clinical visits are likely to receive higher attention. RETAIN was tested on a large health system EHR dataset with 14 million visits completed by 263K patients over an 8 year period and demonstrated predictive accuracy and computational scalability comparable to state-of-the-art methods such as RNN, and ease of interpretability comparable to traditional models.
This paper aims to provide more interpretable models from clinical time series data by relying more on recent time points than distant ones for making predictions. Predictive accuracy is compared to the current state of the art on a set of clinical data.
The problem addressed (model interpretability) is an important one, and certainly a model that a clinician can understand would be preferable to a black-box model with similar accuracy. The assumption that focusing on recent timepoints may help create such models is reasonable on the face of it, but is not well justified in the paper (theoretically or empirically), and it is unclear whether the approach would generalize to any other problems. The lack of confidence in the results and lack of clear application to any other problem limited my enthusiasm. Fundamentally, the paper demonstrates that their approach has similar predictive accuracy to the state of the art on one dataset, but fails to demonstrate that it has any advantages that make it superior despite the similar accuracy.  Key questions that need to be addressed: 1) Handling of irregularly sampled time series: It seems the authors have to assume the same sampling rate for all patients (as on pg 2 they refer to the i and i+1th visit as a time step). However, visits are irregularly spaced and will certainly occur at different frequencies for different patients. How does this affect the algorithm? Similarly, there may be a cluster of frequent visits during an acute illness, but these will likely have redundant information and should be collapsed into a single episode.   2) Why is the assumption of more recent events being most important justified? The entire paper rests on this assumption, but it is not obvious that this should be the case. For example, duration of diabetes is often an important variable for predicting complications, but there we need to understand the onset, not the most recent mention of diabetes in a medical record.  3) A lot of detail on the experiments is missing. It's not clear where the data come from or how this definition of heart failure was created or validated. Further many decisions were made in assigning diagnosis dates (and dates for controls) that are never justified.    4) The evaluation of interpretability contains only anecdotes related to the single experiment. Were any tests done to determine if the proposed approach leads to predictions that are more interpretable by humans? I would think that should be a key evaluation for this paper.  Most concerning, though, is that it's stated that "RETAIN properly puts much attention to the HF-related codes that occurred in recent visits." According to the method for finding HF cases, it seems that in most cases the HF codes should be at or after the dx-date.  It seems that the time series was not appropriately segmented, so that data after diagnosis is being used for the prediction task. Finding that HF codes predict heart failure is not a clinically meaningful model.   5) How well does this method do on other datasets? In particular, are there any experiments on data where ground truth is known? EHRs are very noisy and heart failure diagnosis can be ambiguous in these data. The use of only a single experiment in this case is less convincing.