INTERPOLATION-PREDICTION NETWORKS FOR IRREGULARLY SAMPLED TIME SERIES
In this paper, we present a new deep learning architecture for addressing the problem of supervised learning with sparse and irregularly sampled multivariate time series. The architecture is based on the use of a semi-parametric interpolation network followed by the application of a prediction network. The interpolation network allows for information to be shared across multiple dimensions of a multivariate time series during the interpolation stage, while any standard deep learning model can be used for the prediction network. This work is motivated by the analysis of physiological time series data in electronic health records, which are sparse, irregularly sampled, and multivariate. We investigate the performance of this architecture on both classification and regression tasks, showing that our approach outperforms a range of baseline and recently proposed models.
I have mixed feelings about this submission, and as such, I look forward to discussing it with both the authors and my fellow reviewers. In short, I like the simplicity of the idea, but I am uncertain about the degree to which it satisfies ICLR's novelty criterion ("present substantively new ideas or explore an underexplored or highly novel question"); I do feel confident that some ICLR readers would (perhaps unfairly) describe this approach as "obvious." The paper's presentation suffers, and it fails to communicate essential details clearly. Finally, for folks familiar with healthcare data and MIMIC-III specifically, the results are underwhelming: yes, the proposed approach beats (the authors' own implementations of) baselines, but it underperforms other published results on the MIMIC-III 48-hour mortality task ([1][2] report AUCs of 0.87 or higher). As such, I am assigning the paper a "weak accept" to communicate my ambivalence and reserve the right to adjust it up or down after discussion.
SUMMARY
This paper proposes an "interpolation layer" to resample irregularly sampled time series before feeding them into a neural net architecture. The interpolation layer consists of parametric kernels, e.g., radial basis functions, configured to estimate the values of input time series at reference time points based on univariate temporal and then multivariate correlations. The outputs include smooth and transient interpolated values (controlled by kernel bandwidth) and counts (referred to as intensity) at each reference point. As far as I understand, this model can be trained end-to-end. The paper also proposes a simple strategy for combatting overfitting (add an autoencoder and reconstruction error term to the objective in combination with a heuristic in which some points are masked as inputs and must be interpolated from non-masked points). In experiments on two data sets (UWaveGesture and a medical data set) and two tasks (classification and regression) this approach outperforms the main competing approaches [3][4][5][6] in most contexts.
Below I provide a list of strengths, weaknesses, and general questions or feedback.
STRENGTHS
- I applaud the simplicity of the idea: this much simpler framework leverages many of the intuitions behind the GP adapter framework (GP-GRU) [4][5] with comparable performance and appears to train orders of magnitude faster (caveat: on one data set and task)
- It likewise outperforms both commonly used preprocessing (GRU-F) [2][3] and the much more complicated neural net architecture (GRU-HD) from [6] (across two datasets and tasks)
- The simplicity of this approach probably lends itself to additional customization and innovation
- The literature review seems quite thorough and does an especially nice job of covering recent work on RNNs for multivariate time series and irregular sampling or missing values
- The experiments are thorough and well-designed overall. The authors use two data sets and two tasks (classification and regression). More data sets and tasks is always nice, but even two is pretty laudable (many authors might settle on just one given the experimental and computational effort required for these experiments). They include and beat or outperform two baselines that can justifiably be called state-of-the-art (GP-GRU and GRU-HD).
I think a relatively safe takeaway is that for irregularly sampled data, this approach is is preferable to both heuristic preprocessing and more complex models. That seems like a not insignificant finding in empirical machine learning for messy time series data.
WEAKNESSES
- Section 3 is possibly the most critical section (since it describes the contribution) but is hard to follow: I don't envy the authors the task of explaining a variable with two superscripts and three subscripts (Equation 1), but it IS their paper, so it's on them to do it. See feedback section for other examples.
- Although I consider the related work well done, I can't help but wonder if there isn't older work on RBFs, etc., that might have been missed (I mostly want to encourage the authors to look once more and then come back and tell me I'm wrong).
- The MIMIC-III experiments omit the GP-GRU model, which weakens the results by leaving the reader to imagine how it might compare (I would expect it to outperform the proposed approach by an even wider margin than it did for UWave).
- I am sympathetic to the idea of fixing certain architectural choices, e.g., layers and units in the GRU and number of inducing points, across all models because it (a) gives the appearance of a "fair comparison" and (b) reduces burden of effort, but I do not agree that it yields a truly fair comparison. The GRU-* model performance on UWave is suspiciously bad, suggesting severe overfitting and the possibility that the models are overparameterized. It leaves the reader wondering if the architectural choices happen to be optimal for the proposed model only (whether by accident or design). A truly fair comparison requires independently tuning hyperparameters for each model.
- Although the proposed approach outperforms baselines in these experiments, the overall results are underwhelming in the wider context of recent work using MIMIC-III. Multiple publications have reported AUCS of 0.87 [1][2] or higher for 48-hour risk of mortality (it is difficult to compare the LOS results since different papers use different units). Of course, the experiments use different cohorts and variables so they're not directly comparable, but it nonetheless diminishes the potential impact of the results presented here.
FEEDBACK AND QUESTIONS
- I had to read 3.2.1 multiple times to understand the relationships between the different "layers" in the interpolator, and I'm still not sure what the relationship is between the smooth and transient kernels or exactly how the intensity values are estimated (are they just windowed counts or weighted sums?).
- I'm also not 100% clear on (a) which parameters (if any) in the interpolator are optimized during end-to-end learning and which are just fixed or tuned as hyperparameters. This should be stated clearly and even better, I'd recommend writing down the gradient update rules for the interpolator parameters (you can put them in the appendix).
- Since the model uses global structure for interpolation and requires pre-specifying the number of inducing points, could it be used to make continuous predictions (and how?), e.g., forecast mortality at each hour?
- On a related note, if the number of inducing points is pre-specified, can the model be applied to sequences of different length?
- How does performance depend on choice of number of inducing points?
- How does the proposed approach handle time series that are missing entirely, e.g., if no pH values are measured?
- What does Table 3 in the appendix mean by "missingness?" Given that the paper is concerned with irregular sampling (not missing data), I would expect statistics on sampling rates, not missingness...
- Why derive your own MIMIC-III subset and tasks rather than use one of several pre-existing benchmarks (both of which include more variables and tasks) [1][2]?
- FYI: the Che, et al., 2016, paper on missing values [6] has been published in JBIO, so you should cite that version.
REFERENCES
[1] Purushotham, et al. "Benchmark of Deep Learning Models on Large Healthcare MIMIC Datasets." arXiv preprint arXiv:1710.08531 (2017)
[2] Harutyunyan, et al. "Multitask learning and benchmarking with clinical time series data." arXiv preprint arXiv:1703.07771 (2017)
[3] Lipton, Kale, and Wetzel, 2016
[4] Li and Marlin, 2016.
[5] Futoma, et al., 2017.
[6] Che, et al., 2016. <-- new JBIO 2018 version!