Multivariate time series data in practical applications, such as health care, geoscience, and biology, are characterized by a variety of missing values. In time series prediction and other related tasks, it has been noted that missing values and their missing patterns are often correlated with the target labels, a.k.a., informative missingness. There is very limited work on exploiting the missing patterns for effective imputation and improving prediction performance. In this paper, we develop novel deep learning models, namely GRU-D, as one of the early attempts. GRU-D is based on Gated Recurrent Unit (GRU), a state-of-the-art recurrent neural network. It takes two representations of missing patterns, i.e., masking and time interval, and effectively incorporates them into a deep model architecture so that it not only captures the long-term temporal dependencies in time series, but also utilizes the missing patterns to achieve better prediction results. Experiments of time series classification tasks on real-world clinical datasets (MIMIC-III, PhysioNet) and synthetic datasets demonstrate that our models achieve state-of-the-art performance and provides useful insights for better understanding and utilization of missing values in time series analysis.
1 INTRODUCTION
Multivariate time series data are ubiquitous in many practical applications ranging from health care, geoscience, astronomy, to biology and others. They often inevitably carry missing observations due to various reasons, such as medical events, saving costs, anomalies, inconvenience and so on. It has been noted that these missing values are usually informative missingness (Rubin, 1976), i.e., the missing values and patterns provide rich information about target labels in supervised learning tasks (e.g, time series classification). To illustrate this idea, we show some examples from MIMIC-III, a real world health care dataset in Figure 1. We plot the Pearson correlation coefficient between variable missing rates, which indicates how often the variable is missing in the time series, and the labels of our interests such as mortality and ICD-9 diagnoses. We observe that the missing rate is correlated with the labels, and the missing rates with low rate values are usually highly (either positive or negative) correlated with the labels. These findings demonstrate the usefulness of missingness patterns in solving a prediction task.
In the past decades, various approaches have been developed to address missing values in time series (Schafer & Graham, 2002). A simple solution is to omit the missing data and to perform analysis only on the observed data. A variety of methods have been developed to fill in the missing values, such as smoothing or interpolation (Kreindler & Lumsden, 2012), spectral analysis (Mondal & Percival, 2010), kernel methods (Rehfeld et al., 2011), multiple imputation (White et al., 2011),
and EM algorithm (Garcı́a-Laencina et al., 2010). Schafer & Graham (2002) and references therein provide excellent reviews on related solutions. However, these solutions often result in a twostep process where imputations are disparate from prediction models and missing patterns are not effectively explored, thus leading to suboptimal analyses and predictions (Wells et al., 2013).
In the meantime, Recurrent Neural Networks (RNNs), such as Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) and Gated Recurrent Unit (GRU) (Cho et al., 2014), have shown to achieve the state-of-the-art results in many applications with time series or sequential data, including machine translation (Bahdanau et al., 2014; Sutskever et al., 2014) and speech recognition (Hinton et al., 2012). RNNs enjoy several nice properties such as strong prediction performance as well as the ability to capture long-term temporal dependencies and variable-length observations. RNNs for missing data has been studied in earlier works (Bengio & Gingras, 1996; Tresp & Briegel, 1998; Parveen & Green, 2001) and applied for speech recognition and blood-glucose prediction. Recent works (Lipton et al., 2016; Choi et al., 2015) tried to handle missingness in RNNs by concatenating missing entries or timestamps with the input or performing simple imputations. However, there have not been works which model missing patterns into a systematically modified RNN structure for time series classification problems. Exploiting the power of customized RNN models along with the informativeness of missing patterns is a new promising venue to effectively model multivariate time series and is the main motivation behind our work.
In this paper, we develop a novel deep learning model based on GRU, namely GRU-D, to effectively exploit two representations of informative missingness patterns, i.e., masking and time interval. Masking informs the model which inputs are observed (or missing), while time interval encapsulates the input observation patterns. Our model captures the observations and their dependencies by applying masking and time interval (using a decay term) to the inputs and network states of GRU, and jointly train all model components using back-propagation. Thus, our model not only captures the long-term temporal dependencies of time series observations but also utilizes the missing patterns to improve the prediction results. Empirical experiments on real-world clinical datasets as well as synthetic datasets demonstrate that our proposed model outperforms strong deep learning models built on GRU with imputation as well as other strong baselines. These experiments show that our proposed method is suitable for many time series classification problems with missing data, and in particular is readily applicable to the predictive tasks in emerging health care applications. Moreover, our method provides useful insights into more general research challenges of time series analysis with missing data beyond classification tasks, including 1) a general deep learning framework to handle time series with missing data, 2) effective solutions to characterize the missing patterns of not missing-completely-at-random time series data such as modeling masking and time interval, and 3) an insightful approach to study the impact of variable missingness on the prediction labels by decay analysis.
2 RNN MODELS FOR TIME SERIES WITH MISSING VARIABLES
We denote a multivariate time series with D variables of length T as X = (x1,x2, . . . ,xT ) T ∈ RT×D, where for each t ∈ {1, 2, . . . , T}, xt ∈ RD represents the tth observations (a.k.a., measurements) of all variables and xdt denotes the measurement of dth variable of xt. Let st ∈ R denote the time-stamp when the tth observation is obtained and we assume that the first observation is made at
time-stamp 0 (i.e., s1 = 0). A time series X could have missing values. We introduce a masking vectormt ∈ {0, 1}D to denote which variables are missing at time step t. The masking vector for xt is given by
mdt = { 1, if xdt is observed 0, otherwise
For each variable d, we also maintain the time interval δdt ∈ R since its last observation as
δdt =  st − st−1 + δ d t−1, t > 1,m d t−1 = 0
st − st−1, t > 1,mdt−1 = 1 0, t = 1
An example of these notations is illustrated in Figure 2. In this paper, we are interested in the time series classification problem, where we predict the labels ln given the time series data D, where D = {(Xn, sn,Mn,∆n)}Nn=1, andXn = [ x (n) 1 , . . . ,x (n) Tn ] , sn = [ s (n) 1 , . . . , s (n) Tn ] ,Mn =[
m (n) 1 , . . . ,m (n) Tn ] , ∆n = [ δ (n) 1 , . . . , δ (n) Tn ] , and ln ∈ {1, . . . , L}.
2.1 GRU-RNN FOR TIME SERIES CLASSIFICATION
We investigate the use of recurrent neural networks (RNN) for time-series classification, as their recursive formulation allow them to handle variable-length sequences naturally. Moreover, RNN shares the same parameters across all time steps which greatly reduces the total number of parameters we need to learn. Among different variants of the RNN, we specifically consider an RNN with gated recurrent units (Cho et al., 2014; Chung et al., 2014), but similar discussion and convolutions are also valid for other RNN models such as LSTM (Hochreiter & Schmidhuber, 1997).
The structure of GRU is shown in Figure 3(a). For each jth hidden unit, GRU has a reset gate rjt and an update gate zjt to control the hidden state h j t at each time t. The update functions are shown as follows:
rt = σ (W rxt +U rht−1 + br) zt = σ (W zxt +Uzht−1 + bz)
h̃t = tanh (Wxt +U(rt ht−1) + b) ht = (1− zt) ht−1 + zt h̃t where matricesW z,W r,W ,Uz,U r,U and vectors bz, br, b are model parameters. We use σ for element-wise sigmoid function, and for element-wise multiplication. This formulation assumes that all the variables are observed. A sigmoid or soft-max layer is then applied on the output of the GRU layer at the last time step for classification task.
Existing work on handling missing values lead to three possible solutions with no modification on GRU network structure. One straightforward approach is simply replacing each missing observation
with the mean of the variable across the training examples. In the context of GRU, we have
xdt ← mdtxdt + (1−mdt )x̃d (1)
where x̃d = ∑N n=1 ∑Tn t=1m d t,nx d t,n /∑N n=1 ∑Tn t=1m d t,n. We refer to this approach as GRU-mean.
A second approach is exploiting the temporal structure in time series. For example, we may assume any missing value is same as its last measurement and use forward imputation (GRU-forward), i.e.,
xdt ← mdtxdt + (1−mdt )xdt′ (2)
where t′ < t is the last time the d-th variable was observed.
Instead of explicitly imputing missing values, the third approach simply indicates which variables are missing and how long they have been missing as a part of input, by concatenating the measurement, masking and time interval vectors as
x (n) t ← [ x (n) t ;m (n) t ; δ (n) t ] (3)
where x(n)t can be either from Equation (1) or (2). We later refer to this approach as GRU-simple.
Several recent works (Lipton et al., 2016; Choi et al., 2015; Pham et al., 2016) use RNNs on EHR data to model diseases and to predict patient diagnosis from health care time series data with irregular time stamps or missing values, but none of them have explicitly attempted to capture and utilize the missing patterns into their RNNs via systematically modified network architectures. Choi et al. (2015) feeds medical codes along with its time stamps into GRU model to predict the next medical event. This feeding time stamps idea is equivalent to the baseline GRU-simple without feeding the masking, which we denote as GRU-simple (interval only). Pham et al. (2016) takes time stamps into LSTM model, and modify its forgetting gate by either time decay and parametric time both from time stamps. However, their non-trainable decay is not that flexible, and the parametric time also does not change RNN model structure and is similar to GRU-simple (interval only). In addition, neither of them consider missing values in time series medical records, and the time stamp input used in these two models is vector for one patient, but not matrix for each input variable of one patient as ours. Lipton et al. (2016) achieves their best performance on diagnosis prediction by feeding masking with zero-filled missing values. Their model is equivalent to GRU-simple without feeding the time interval, and no model structure modification is made for further capturing and utilizing missingness. We denote their best model as GRU-simple (masking only). Conclusively, our GRU-simple baseline can be considered as a generalization from all related RNN models mentioned above and as shown in the experiments these GRU-simple variations have quite close performance.
These approaches solve the missing value issue to a certain extent, However, it is known that imputing the missing value with mean or forward imputation cannot distinguish whether missing values are imputed or truly observed. Simply concatenating masking and time interval vectors fails to exploit the temporal structure of missing values. Thus none of them fully utilize missingness in data to achieve desirable performance.
2.2 GRU-D: MODEL WITH TRAINABLE DECAYS
To fundamentally address the issue of missing values in time series, we notice two important properties of the missing values in time series, especially in health care domains: First, the value of the missing variable tend to be close to some default value if its last observation happens a long time ago. This property usually exists in health care data for human body as homeostasis mechanisms and is considered to be critical for disease diagnosis and treatment (Vodovotz et al., 2013). Second, the influence of the input variables will fade away over time if the variable has been missing for a while. For example, one medical feature in electronic health records (EHRs) is only significant in a certain temporal context (Zhou & Hripcsak, 2007). Therefore we propose a GRU-based model called GRU-D, in which a decay mechanism is designed for the input variables and the hidden states to capture the aforementioned properties. We introduce decay rates in the model to control the decay mechanism by considering the following important factors. First, each input variable in health care time series has its own medical meaning and importance. The decay rates should be flexible to differ from variable to variable based on the underlying properties associated with the variables.
Second, as we see lots of missing patterns are informative in prediction tasks, the decay rate should be indicative of such patterns and benefits the prediction tasks. Furthermore, since the missing patterns are unknown and possibly complex, we aim at learning decay rates from the training data rather than being fixed a priori. That is, we model a vector of decay rates γ as
γt = exp {−max (0,W γδt + bγ)} (4) where W γ and bγ are model parameters that we train jointly with all the other parameters of the GRU. We chose the exponentiated negative rectifier in order to keep each decay rate monotonically decreasing in a reasonable range between 0 and 1. Note that other formulations such as a sigmoid function can be used instead, as long as the resulting decay is monotonic and is in the same range.
Our proposed GRU-D model incorporates two different trainable decays to utilize the missingness directly with the input feature values and implicitly in the RNN states. First, for a missing variable, we use an input decay γx to decay it over time toward the empirical mean (which we take as a default configuration), instead of using the last observation as it is. Under this assumption, the trainable decay scheme can be readily applied to the measurement vector by
xdt ← mdtxdt + (1−mdt )γxdtxdt′ + (1−mdt )(1− γxdt )x̃d (5)
where xdt′ is the last observation of the d-th variable (t ′ < t) and x̃d is the empirical mean of the dth variable. When decaying the input variable directly, we constrain W γx to be diagonal, which effectively makes the decay rate of each variable independent from the others. Sometimes the input decay may not fully capture the missing patterns since not all missingness information can be represented in decayed input values. In order to capture richer knowledge from missingness, we also have a hidden state decay γh in GRU-D. Intuitively, this has an effect of decaying the extracted features (GRU hidden states) rather than raw input variables directly. This is implemented by decaying the previous hidden state ht−1 before computing the new hidden state ht as
ht−1 ← γht ht−1, (6) in which case we do not constrain W γh to be diagonal. In addition, we feed the masking vectors (mt) directly into the model. The update functions of GRU-D are
zt = σ (W zxt +Uzht−1 + V zmt + bz) rt = σ (W rxt +U rht−1 + V rmt + br)
h̃t = tanh (Wxt +U(rt ht−1) + Vmt + b) ht = (1− zt) ht−1 + zt h̃t where xt and ht−1 are respectively updated by Equation (5) and (6), and V z,V r,V are new parameters for masking vectormt.
To validate GRU-D model and demonstrate how it utilizes informative missing patterns, in Figure 4, we show the input decay (γx) plots and hidden decay (γh) histograms for all the variables for predicting mortality on PhysioNet dataset. For input decay, we notice that the decay rate is almost constant for the majority of variables. However, a few variables have large decay which means that the model relies less on the previous observations for prediction. For example, the changes in the variable values of weight, arterial pH, temperature, and respiration rate are known to impact the ICU patients health condition. The hidden decay histograms show the distribution of decay parameters related to each variable. We noticed that the parameters related to variables with smaller missing rate are more spread out. This indicates that the missingness of those variables has more impact on decaying or keeping the hidden states of the models.
Notice that the decay term can be generalized to LSTM straightforwardly. In practical applications, missing values in time series may contain useful information in a variety of ways. A better model should have the flexibility to capture different missing patterns. In order to demonstrate the capacity of our GRU-D model, we discuss some model variations in Appendix A.2.
3 EXPERIMENTS
3.1 DATASET DESCRIPTIONS AND EXPERIMENTAL DESIGN
We demonstrate the performance of our proposed models on one synthetic and two real-world health-care datasets1 and compare it to several strong machine learning and deep learning approaches
1A summary statistics of the three datasets is shown in Appendix A.3.1.
6: Cholesterol mr: 0.9989 27: TroponinI mr: 0.9984 28: TroponinT mr: 0.9923 3: Albumin mr: 0.9915 0: ALP mr: 0.9888 1: ALT mr: 0.9885 2: AST mr: 0.9885 5: Bilirubin mr: 0.9884 16: Lactate mr: 0.9709 24: SaO2 mr: 0.9705 30: WBC mr: 0.9532
11: Glucose mr: 0.9528 19: Na mr: 0.9508 18: Mg mr: 0.9507 12: HCO3 mr: 0.9507 4: BUN mr: 0.9496 7: Creatinine mr: 0.9493 22: Platelets mr: 0.9489 15: K mr: 0.9477 13: HCT mr: 0.9338 21: PaO2 mr: 0.9158 20: PaCO2 mr: 0.9157
32: pH mr: 0.9118 9: FiO2 mr: 0.883 23: RespRate mr: 0.8053 10: GCS mr: 0.7767 26: Temp mr: 0.6915 31: Weight mr: 0.5452 29: Urine mr: 0.5095 17: MAP mr: 0.2141 8: DiasABP mr: 0.2054 25: SysABP mr: 0.2052 14: HR mr: 0.1984
(a) x-axis, time interval δdt between 0 and 24 hours; y-axis, value of decay rate γxdt between 0 and 1.
101
102
6: Cholesterol mr: 0.9989 27: TroponinI mr: 0.9984 28: TroponinT mr: 0.9923 3: Albumin mr: 0.9915 0: ALP mr: 0.9888 1: ALT mr: 0.9885 2: AST mr: 0.9885 5: Bilirubin mr: 0.9884 16: Lactate mr: 0.9709 24: SaO2 mr: 0.9705 30: WBC mr: 0.9532
101
102
11: Glucose mr: 0.9528 19: Na mr: 0.9508 18: Mg mr: 0.9507 12: HCO3 mr: 0.9507 4: BUN mr: 0.9496 7: Creatinine mr: 0.9493 22: Platelets mr: 0.9489 15: K mr: 0.9477 13: HCT mr: 0.9338 21: PaO2 mr: 0.9158 20: PaCO2 mr: 0.9157
0.3 0.3
101
102
32: pH mr: 0.9118
0.3 0.3
9: FiO2 mr: 0.883
0.3 0.3
23: RespRate mr: 0.8053
0.3 0.3
10: GCS mr: 0.7767
0.3 0.3
26: Temp mr: 0.6915
0.3 0.3
31: Weight mr: 0.5452
0.3 0.3
29: Urine mr: 0.5095
0.3 0.3
17: MAP mr: 0.2141
0.3 0.3
8: DiasABP mr: 0.2054
0.3 0.3
25: SysABP mr: 0.2052
0.3 0.3
14: HR mr: 0.1984
(b) x-axis, value of decay parameter W γh ; y-axis, count.
Figure 4: Plots of input decay γxt (top) and histrograms of hidden state decay γht (bottom) of all 33 variables in GRU-D model for predicting mortality on PhysioNet dataset. Variables in green are lab measurements; variables in red are vital signs; mr refers to missing rate.
in classification tasks. We evaluate our models for different settings such as early prediction and different training sizes and investigate the impact of informative missingness.
Gesture phase segmentation dataset (Gesture) This UCI dataset (Madeo et al., 2013) has multivariate time series features, regularly sampled and with no missing values, for 5 different gesticulations. We extracted 378 time series and generate 4 synthetic datasets for the purpose of understanding model behaviors with different missing patterns. We treat it as multi-class classification task.
PhysioNet Challenge 2012 dataset (PhysioNet) This dataset, from PhysioNet Challenge 2012 (Silva et al., 2012), is a publicly available collection of multivariate clinical time series from 8000 intensive care unit (ICU) records. Each record is a multivariate time series of roughly 48 hours and contains 33 variables such as Albumin, heart-rate, glucose etc. We used Training Set A subset in our experiments since outcomes (such as in-hospital mortality labels) are publicly available only for this subset. We conduct the following two prediction tasks on this dataset: 1) Mortality task: Predict whether the patient dies in the hospital. There are 554 patients with positive mortality label. We treat this as a binary classification problem. and 2) All 4 tasks: Predict 4 tasks: in-hospital mortality, length-of-stay less than 3 days, whether the patient had a cardiac condition, and whether the patient was recovering from surgery. We treat this as a multi-task classification problem.
MIMIC-III dataset (MIMIC-III) This public dataset (Johnson et al., 2016) has deidentified clinical care data collected at Beth Israel Deaconess Medical Center from 2001 to 2012. It contains over 58,000 hospital admission records. We extracted 99 time series features from 19714 admission records for 4 modalities including input-events (fluids into patient, e.g., insulin), output-events (fluids out of the patient, e.g., urine), lab-events (lab test results, e.g., pH values) and prescription-events (drugs prescribed by doctors, e.g., aspirin). These modalities are known to be extremely useful for monitoring ICU patients. We only use the first 48 hours data after admission from each time series. We perform following two predictive tasks: 1) Mortality task: Predict whether the patient dies in the hospital after 48 hours. There are 1716 patients with positive mortality label and we perform binary classification. and 2) ICD-9 Code tasks: Predict 20 ICD-9 diagnosis categories (e.g., respiratory system diagnosis) for each admission. We treat this as a multi-task classification problem.
3.2 METHODS AND IMPLEMENTATION DETAILS
We categorize all evaluated prediction models into three following groups:
• Non-RNN Baselines (Non-RNN): We evaluate logistic regression (LR), support vector machines (SVM) and Random Forest (RF) which are widely used in health care applications. • RNN Baselines (RNN): We take GRU-mean, GRU-forward, GRU-simple, and LSTM-mean
(LSTM model with mean-imputation on the missing measurements) as RNN baselines. • Proposed Methods (Proposed): This is our proposed GRU-D model from Section 2.2.
The non-RNN baselines cannot handle missing data directly. We carefully design experiments for nonRNN models to capture the informative missingness as much as possible to have fair comparison with the RNN methods. Since non-RNN models only work with fixed length inputs, we regularly sample the time-series data to get a fixed length input and perform imputation to fill in the missing values. Similar to RNN baselines, we can concatenate the masking vector along with the measurements and feed it to non-RNN models. For PhysioNet dataset, we sample the time series on an hourly basis and propagate measurements forward (or backward) in time to fill gaps. For MIMIC-III dataset, we consider two hourly samples (in the first 48 hours) and do forward (or backward) imputation. Our preliminary experiments showed 2-hourly samples obtains better performance than one-hourly samples for MIMIC-III. We report results for both concatenation of input and masking vectors (i.e., SVM/LR/RF-simple) and only input vector without masking (i.e., SVM/LR/RF-forward). We use the scikit-learn (Pedregosa et al., 2011) for the non-RNN model implementation and tune the parameters by cross-validation. We choose RBF kernel for SVM since it performs better than other kernels.
For RNN models, we use a one layer RNN to model the sequence, and then apply a soft-max regressor on top of the last hidden state hT to do classification. We use 100 and 64 hidden units in GRU-mean for MIMIC-III and PhysioNet datasets, respectively. All the other RNN models were constructed to have a comparable number of parameters.2 For GRU-simple, we use mean imputation for input as shown in Equation (1). Batch normalization (Ioffe & Szegedy, 2015) and dropout (Srivastava et al., 2014) of rate 0.5 are applied to the top regressor layer. We train all the RNN models with the Adam optimization method (Kingma & Ba, 2014) and use early stopping to find the best weights on the validation dataset. All the input variables are normalized to be 0 mean and 1 standard deviation. We report the results from 5-fold cross validation in terms of area under the ROC curve (AUC score). We provide more detailed comparisons of RNN baselines and variations in Appendix A.3.4 and evaluations on multilayer RNN models in Appendix A.3.5.
3.3 QUANTITATIVE RESULTS
Exploiting informative missingness on synthetic dataset As illustrated in Figure 1, missing patterns can be useful in solving prediction tasks. A robust model should exploit informative missingness properly and avoid inducing nonexistent relations between missingness and predictions. To evaluate the impact of modeling missingness we conduct experiments on the synthetic Gesture datasets. We process the data in 4 different settings with the same missing rate but different correlations between missing rate and the label. A higher correlation implies more informative missingness. Figure 5 shows the AUC score comparison of three GRU baseline models (GRU-mean, GRU-forward, GRU-simple) and the proposed GRU-D. Since GRU-mean and GRU-forward do not utilize any missingness (i.e., masking or time interval), they perform similarly across all 4 settings. GRU-simple and GRU-D benefit from utilizing the missingness, especially when the correlation is high. Our GRU-D achieves the best performance in all settings, while GRU-simple fails when the correlation is low. The results on synthetic datasets demonstrates that our proposed model can model and distinguish useful missing patterns in data properly compared with baselines.
Prediction task evaluation on real datasets We evaluate all methods in Section 3.2 on MIMIC-III and PhysioNet datasets. We noticed that dropout in the recurrent layer helps a lot for all RNN models
2Appendix A.3.2 compares all GRU models tested in the experiments in terms of model size.
on both of the datasets, probably because they contain more input variables and training samples than synthetic dataset. Similar to Gal (2015), we apply dropout rate of 0.3 with same dropout samples at each time step on weights W ,U ,V . Table 2 shows the prediction performance of all the models on mortality task. All models except for random forest improve their performance when they feed missingness indicators along with inputs. The proposed GRU-D achieves the best AUC score on both datasets. We also conduct multi-task classification experiments for all 4 tasks on PhysioNet and 20 ICD-9 code tasks on MIMIC-III using all the GRU models. As shown in Table 1, GRU-D performs best in terms of average AUC score across all tasks and in most of the single tasks.
3.4 DISCUSSIONS
Online prediction in early stage Although our model is trained on the first 48 hours data and makes prediction at the last time step, it can be used directly to make predictions before it sees all the time series and can make predictions on the fly. This is very useful in applications such as health care, where early decision making is beneficial and critical for patient care. Figure 6 shows the online prediction results for MIMIC-III mortality task. As we can see, AUC is around 0.7 at first 12 hours for all the GRU models and it keeps increasing when longer time series is fed into these models. GRU-D and GRU-simple, which explicitly handle missingness, perform consistently better than the other two methods. In addition, GRU-D outperforms GRU-simple when making predictions given time series of more than 24 hours, and has at least 2.5% higher AUC score after 30 hours. This indicates that GRU-D is able to capture and utilize long-range temporal missing patterns. Furthermore, GRU-D achieves similar prediction performance (i.e., same AUC) as best non-RNN baseline model with less time series data. As shown in the figure, GRU-D has same AUC performance at 36 hours as the best non-RNN baseline model (RF-simple) at 48 hours. This 12 hour improvement of GRU-D over non-RNN baseline is highly significant in hospital settings such as ICU where time-saving critical decisions demands accurate early predictions.
Model Scalability with growing data size In many practical applications, model scalability with large dataset size is very important. To evaluate the model performance with different training dataset size, we subsample three smaller datasets of 2000 and 10000 admissions from the entire MIMIC-III dataset while keeping the same mortality rate. We compare our proposed models with all GRU baselines and two most competitive non-RNN baselines (SVM-simple, RF-simple) and shows the prediction results in Figure 7. We observe that all models can achieve improved performance given more training samples. However, the improvements of non-RNN baselines are quite limited compared to GRU models, and our GRU-D model achieves the best results on the two larger datasets. These results indicate the performance gap between GRU-D and non-RNN baselines will continue to grow as more data become available.
4 SUMMARY
In this paper, we proposed novel GRU-based model to effectively handle missing values in multivariate time series data. Our model captures the informative missingness by incorporating masking and time interval directly inside the GRU architecture. Empirical experiments on both synthetic and real-world health care datasets showed promising results and provided insightful findings. In our future work, we will explore deep learning approaches to characterize missing-not-at-random data and we will conduct theoretical analysis to understand the behaviors of existing solutions for missing values.
A APPENDIX
A.1 INVESTIGATION OF RELATION BETWEEN MISSINGNESS AND LABELS
In many time series applications, the pattern of missing variables in the time series is often informative and useful for prediction tasks. Here, we empirically confirm this claim on real health care dataset by investigating the correlation between the missingness and prediction labels (mortality and ICD9 diagnosis categories). We denote the missing rate for a variable d as pdX and calculate it by pdX = 1− 1T ∑T t=1m d t . Note that p d X is dependent on mask vector (m d t ) and number of time steps T . For each prediction task, we compute the Pearson correlation coefficient between pdX and label ` across all the time series. As shown in Figure 1, we observe that on MIMIC-III dataset the missing rates with low rate values are usually highly (either positive or negative) correlated with the labels. The distinct correlation between missingness and labels demonstrates usefulness of missingness patterns in solving prediction tasks.
A.2 GRU-D MODEL VARIATIONS
In this section, we will discuss some variations of GRU-D model, and also compare some related RNN models which are used for time series with missing data with the proposed model.
A.2.1 GRU MODEL WITH DIFFERENT TRAINABLE DECAYS
The proposed GRU-D applies trainable decays on both input and hidden state transitions in order to capture the temporal missing patterns explicitly. This decay idea can be straightforwardly generated to other parts inside the GRU models separately or jointly, given different assumptions on the impact of missingness. As comparisons, we also describe and evaluate several modifications of GRU-D model.
GRU-DI (Figure 8(a)) and GRU-DS (Figure 8(b)) decay only the input and only the hidden state by Equation (5) and (6), respectively. They can be considered as two simplified models of the proposed GRU-D. GRU-DI aims at capturing direct impact of missing values in the data, while GRU-DS captures more indirect impact of missingness. Another intuition comes from this perspective: if an input variable is just missing, we should pay more attention to this missingness; however, if an variable has been missing for a long time and keeps missing, the missingness becomes less important. We can utilize this assumption by decaying the masking. This brings us the model GRU-DM shown in Figure 8(c), where we replace the masking mdt fed into GRU-D in by
mdt ← mdt + (1−mdt )γmdt (1−mdt ) = mdt + (1−mdt )γmdt (7)
where the equality holds since mdt is either 0 or 1. We decay the masking for each variable independently from others by constrainingW γm to be diagonal.
A.2.2 GRU-IMP: GOAL-ORIENTED IMPUTATION MODEL
We may alternatively let the GRU-RNN predict the missing values in the next timestep on its own. When missing values occur only during test time, we simply train the model to predict the measurement vector of the next time step as a language model (Mikolov et al., 2010) and use it to fill the missing values during test time. This is unfortunately not applicable for some time series applications such as in health care domains, which also have missing data during training.
Instead, we propose goal-oriented imputation model here called GRU-IMP, and view missing values as latent variables in a probabilistic graphical model. Given a timeseriesX , we denote all the missing variables by MX and all the observed ones by OX . Then, training a time-series classifier with missing variables becomes equivalent to maximizing the marginalized log-conditional probability of a correct label l, i.e., log p(l|OX). The exact marginalized log-conditional probability is however intractable to compute, and we instead maximize its lowerbound:
log p(l|OX) = log ∑ MX p (l|MX ,OX) p (MX |OX) ≥ EMX∼p(MX |OX) log p (l|MX ,OX)
where we assume the distribution over the missing variables at each time step is only conditioned on all the previous observations:
p (MX |OX) = T∏ t=1 mdt=1∏ 1≤d≤D p(xdt |x1:(t−1),m1:(t−1), δ1:(t−1)) (8)
Although this lowerbound is still intractable to compute exactly, we can approximate it by Monte Carlo method, which amounts to sampling the missing variables at each time as the RNN reads the input sequence from the beginning to the end, such that
xdt ← mdtxdt + (1−mdt )x̃dt (9)
where x̃t ∼ xdt |x1:(t−1),m1:(t−1), δ1:(t−1). By further assuming that x̃t ∼ N ( µt,σ 2 t ) , µt = γt (W xht−1 + bx) and σt = 1, we can use a reparametrization technique widely used in stochastic variational inference (Kingma & Welling, 2013; Rezende et al., 2014) to estimate the gradient of the lowerbound efficiently. During the test time, we simply use the mean of the missing variable, i.e., x̃t = µt, as we have not seen any improvement from Monte Carlo approximation in our preliminary experiments. We view this approach as a goal-oriented imputation method and show its structure in Figure 8(d). The whole model is trained to minimize the classification cross-entropy error `log loss and we take the negative log likelihood of the observed values as a regularizer.
` = `log loss + λ 1
N N∑ n=1 1 Tn Tn∑ t=1 ∑D d=1m d t · log p(xdt |µdt , σdt )∑D d=1m d t
(10)
A.3 SUPPLEMENTARY EXPERIMENT DETAILS
A.3.1 DATA STATISTICS
For each of the three datasets used in our experiments, we list the number of samples, the number of input variables, the mean and max number of time steps for all the samples, and the mean of all the variable missing rates in Table 3.
A.3.2 GRU MODEL SIZE COMPARISON
In order to fairly compare the capacity of all GRU-RNN models, we build each model in proper size so they share similar number of parameters. Table 4 shows the statistics of all GRU-based models for on three datasets. We show the statistics for mortality prediction on the two real datasets, and it’s almost the same for multi-task classifications tasks on these datasets. In addition, having comparable number of parameters also makes all the models have number of iterations and training time close in the same scale in all the experiments.
A.3.3 MULTI-TASK PREDICTION DETAILS
The RNN models for multi-task learning with m tasks is almost the same as that for binary classification, except that 1) the soft-max prediction layer is replaced by a fully connected layer with n sigmoid logistic functions, and 2) a data-driven prior regularizer (Che et al., 2015), parameterized by comorbidity (co-occurrence) counts in training data, is applied to the prediction layer to improve the classification performance. We show the AUC scores for predicting 20 ICD-9 diagnosis categories on MIMIC-III dataset in Figure 9, and all 4 tasks on PhysioNet dataset in Figure 10. The proposed GRU-D achieves the best average AUC score on both datasets and wins 11 of the 20 ICD-9 prediction tasks.
A.3.4 EMPIRICAL COMPARISON OF MODEL VARIATIONS
As a thorough empirical comparison, we test all GRU model variations mentioned in Appendix A.2 along with the proposed GRU-D. These include 1) 4 models with trainable decays (GRU-DI, GRU-DS, GRU-DM, GRU-IMP), and 2) two models simplified from GRU-simple (interval only and masking only). The results are shown in Table 5. As we can see, GRU-D performs best among these models.
A.3.5 EVALUATION ON MULTI-LAYER RNNS
We also conducted experiments on 2-layer RNN models to demonstrate the superiority of our proposed GRU-D models can be generalized to multi-layer RNNs. For all baseline and proposed GRU models, we add one standard GRU layer on top of the baseline or proposed GRU layer. We tested models both with similar number of parameters to single layer models and with more parameters. As shown in Table 6 and 7, our GRU-D model consistently outperforms other baselines in all cases, and models with moderate size perform as good as larger models with more parameters. Compared with 1-layer RNNs, all models with deeper structures perform much better on the large MIMIC-III dataset but no better on the relative small PhysioNet dataset.

This paper presents a modified gated RNN caled GRU-D that deals with time series which display a lot of missing values in their input. They work on two fronts. The first deals with the missing inputs directly by using a learned convex combination of the previous available value (forward imputation) and the mean value (mean imputation). The second includes dampening the recurrent layer not unlike a second reset gate, but parametrized according to the time elapsed since the last available value of each attributes.
Positives
------------
- Clear definition of the task (handling missing values for classification of time series)
- Many interesting baselines to test the new model against.
- The model presented deals with the missing values in a novel, ML-type way (learn new dampening parameters).
- The extensive tests done on the datasets is probably the greatest asset of this paper.
Negatives
-------------
- The paper could use some double checking for typos.
- The Section A.2.3 really belongs in the main article as it deals with important related works. Swap it with the imprecise diagrams of the model if you need space.
- No mention of any methods from the statistics litterature.
Here are the two main points of this review that informs my decision:
1. The results, while promising, are below expectations. The paper hasn’t been able to convince me that GRU-simple (without intervals) isn’t just as well-suited for the task of handling missing inputs as GRU-D. In the main paper, GRU-simple is presented as the main baseline. Yet, it includes a lot of extraneous parameters (the intervals) that, according to Table 5, probably hurts the model more than it helps it. Having a third of it’s parameters being of dubious value, it brings the question of the fairness of the comparison done in the main paper, especially since in the one table where GRU-simple (without intervals) is present, GRU-D doesn’t significantly outperforms it.
2. My second concern, and biggest, is with some claims that are peppered through the paper. The first is about the relationship with the presence rate of data in the dataset and the diagnostics. I might be wrong, but that only indicates that the doctor in charge of that patient requested the relevant analyses be done according to the patient’s condition. That would mean that an expert system based on this data would always seem to be one step behind.
The second claim is the last sentence of the introduction, which sets huge expectations that were not met by the paper. Another is that “simply concatenating masking and time interval vectors fails to exploit the temporal structure of missing values” is unsubstantiated and actually disproven later in the paper.
Yet another is the conclusion that since GRU models displayed the best improvement between a subsample of the dataset and the whole of it means that the improvement is going to continue to grow as more data is added. This fails to consider that non-GRU models actually started with much better results than most GRU ones.
Lastly is their claim to capture informative missingness by incorporating masking and time intervals directly inside the GRU architecture. While the authors did make these changes, the fact that they also concatenate the mask to the input, just like GRU-simple (without intervals), leads me to question the actual improvement made by GRU-D.
Given that, while I find that the work that has been put into the paper is above average, I wouldn’t accept that paper without a reframing of the findings and a better focus on the real contribution of this paper, which I believe is the novel way to parametrize the choice of imputation method.


SPEECH RECOGNITION SYSTEM
This paper presents a simple end-to-end model for speech recognition, combining a convolutional network based acoustic model and a graph decoding. It is trained to output letters, with transcribed speech, without the need for force alignment of phonemes. We introduce an automatic segmentation criterion for training from sequence annotation without alignment that is on par with CTC (Graves et al., 2006) while being simpler. We show competitive results in word error rate on the Librispeech corpus (Panayotov et al., 2015) with MFCC features, and promising results from raw waveform.
1 INTRODUCTION
We present an end-to-end system to speech recognition, going from the speech signal (e.g. MelFrequency Cepstral Coefficients (MFCC), power spectrum, or raw waveform) to the transcription. The acoustic model is trained using letters (graphemes) directly, which take out the need for an intermediate (human or automatic) phonetic transcription. Indeed, the classical pipeline to build state of the art systems for speech recognition consists in first training an HMM/GMM model to force align the units on which the final acoustic model operates (most often context-dependent phone states). This approach takes its roots in HMM/GMM training (Woodland & Young, 1993). The improvements brought by deep neural networks (DNNs) (Mohamed et al., 2012; Hinton et al., 2012) and convolutional neural networks (CNNs) (Sercu et al., 2015; Soltau et al., 2014) for acoustic modeling only extend this training pipeline.
The current state of the art on Librispeech (the dataset that we used for our evaluations) uses this approach too (Panayotov et al., 2015; Peddinti et al., 2015b), with an additional step of speaker adaptation (Saon et al., 2013; Peddinti et al., 2015a). Recently, Senior et al. (2014) proposed GMMfree training, but the approach still requires to generate a force alignment. An approach that cut ties with the HMM/GMM pipeline (and with force alignment) was to train with a recurrent neural network (RNN) (Graves et al., 2013) for phoneme transcription. There are now competitive end-to-end approaches of acoustic models toppled with RNNs layers as in (Hannun et al., 2014; Miao et al., 2015; Saon et al., 2015; Amodei et al., 2015), trained with a sequence criterion (Graves et al., 2006). However these models are computationally expensive, and thus take a long time to train.
Compared to classical approaches that need phonetic annotation (often derived from a phonetic dictionary, rules, and generative training), we propose to train the model end-to-end, using graphemes directly. Compared to sequence criterion based approaches that train directly from speech signal to graphemes (Miao et al., 2015), we propose a simple(r) architecture (23 millions of parameters for our best model, vs. 100 millions of parameters in (Amodei et al., 2015)) based on convolutional networks
for the acoustic model, toppled with a graph transformer network (Bottou et al., 1997), trained with a simpler sequence criterion. Our word-error-rate on clean speech is slightly better than (Hannun et al., 2014), and slightly worse than (Amodei et al., 2015), in particular factoring that they train on 12,000 hours while we only train on the 960h available in LibriSpeech’s train set. Finally, some of our models are also trained on the raw waveform, as in (Palaz et al., 2013; 2015; Sainath et al., 2015). The rest of the paper is structured as follows: the next section presents the convolutional networks used for acoustic modeling, along with the automatic segmentation criterion. The following section shows experimental results comparing different features, the criterion, and our current best word error rates on LibriSpeech.
2 ARCHITECTURE
Our speech recognition system is a standard convolutional neural network (LeCun & Bengio, 1995) fed with various different features, trained through an alternative to the Connectionist Temporal Classification (CTC) (Graves et al., 2006), and coupled with a simple beam search decoder. In the following sub-sections, we detail each of these components.
2.1 FEATURES
We consider three types of input features for our model: MFCCs, power-spectrum, and raw wave. MFCCs are carefully designed speech-specific features, often found in classical HMM/GMM speech systems (Woodland & Young, 1993) because of their dimensionality compression (13 coefficients are often enough to span speech frequencies). Power-spectrum features are found in most recent deep learning acoustic modeling features (Amodei et al., 2015). Raw wave has been somewhat explored in few recent work (Palaz et al., 2013; 2015). ConvNets have the advantage to be flexible enough to be used with either of these input feature types. Our acoustic models output letter scores (one score per letter, given a dictionary L).
2.2 CONVNET ACOUSTIC MODEL
The acoustic models we considered in this paper are all based on standard 1D convolutional neural networks (ConvNets). ConvNets interleave convolution operations with pointwise non-linearity operations. Often ConvNets also embark pooling layers: these type of layers allow the network to “see” a larger context, without increasing the number of parameters, by locally aggregating the previous convolution operation output. Instead, our networks leverage striding convolutions. Given (xt)t=1...Tx an input sequence with Tx frames of dx dimensional vectors, a convolution with kernel width kw, stride dw and dy frame size output computes the following:
yit = bi + dx∑ j=1 kw∑ k=1 wi,j,k x j dw×(t−1)+k ∀1 ≤ i ≤ dy, (1)
where b ∈ Rdy and w ∈ Rdy×dx×kw are the parameters of the convolution (to be learned).
Pointwise non-linear layers are added after convolutional layers. In our experience, we surprisingly found that using hyperbolic tangents, their piecewise linear counterpart HardTanh (as in (Palaz et al., 2015)) or ReLU units lead to similar results.
There are some slight variations between the architectures, depending on the input features. MFCC-based networks need less striding, as standard MFCC filters are applied with large strides on the input
raw sequence. With power spectrum-based and raw wave-based networks, we observed that the overall stride of the network was more important than where the convolution with strides were placed. We found thus preferrable to set the strided convolutions near the first input layers of the network, as it leads to the fastest architectures: with power spectrum features or raw wave, the input sequences are very long and the first convolutions are thus the most expensive ones.
The last layer of our convolutional network outputs one score per letter in the letter dictionary (dy = |L|). Our architecture for raw wave is shown in Figure 1 and is inspired by (Palaz et al., 2015). The architectures for both power spectrum and MFCC features do not include the first layer. The full network can be seen as a non-linear convolution, with a kernel width of size 31280 and stride equal to 320; given the sample rate of our data is 16KHz, label scores are produced using a window of 1955 ms, with steps of 20ms.
2.3 INFERRING SEGMENTATION WITH AUTOSEGCRITERION
Most large labeled speech databases provide only a text transcription for each audio file. In a classification framework (and given our acoustic model produces letter predictions), one would need the segmentation of each letter in the transcription to train properly the model. Unfortunately, manually labeling the segmentation of each letter would be tedious. Several solutions have been explored in the speech community to alleviate this issue: HMM/GMM models use an iterative EM procedure: (i) during the Estimation step, the best segmentation is inferred, according to the current model, by maximizing the joint probability of the letter (or any sub-word unit) transcription and input sequence. (ii) During the Maximization step the model is optimized by minimizing a frame-level criterion, based on the (now fixed) inferred segmentation. This approach is also often used to boostrap the training of neural network-based acoustic models.
Other alternatives have been explored in the context of hybrid HMM/NN systems, such as the MMI criterion (Bahl et al., 1986) which maximizes the mutual information between the acoustic sequence and word sequences or the Minimum Bayse Risk (MBR) criterion (Gibson & Hain, 2006).
More recently, standalone neural network architectures have been trained using criterions which jointly infer the segmentation of the transcription while increase the overall score of the right transcription (Graves et al., 2006; Palaz et al., 2014). The most popular one is certainly the Connectionist Temporal Classification (CTC) criterion, which is at the core of Baidu’s Deep Speech architecture (Amodei et al., 2015). CTC assumes that the network output probability scores, normalized at the frame level. It considers all possible sequence of letters (or any sub-word units), which can lead to a to a given transcription. CTC also allow a special “blank” state to be optionally inserted between each letters. The rational behind the blank state is two-folds: (i) modeling “garbage” frames which might occur between each letter and (ii) identifying the separation between two identical consecutive letters in a transcription. Figure 2a shows an example of the sequences accepted by CTC for a given transcription. In practice, this graph is unfolded as shown in Figure 2b, over the available frames output by the acoustic model. We denote Gctc(θ, T ) an unfolded graph over T frames for a given transcription θ, and π = π1, . . . , πT ∈ Gctc(θ, T ) a path in this graph representing a (valid) sequence of letters for this transcription. At each time step t, each node of the graph is assigned with the corresponding log-probability letter (that we denote ft(·)) output by the acoustic model. CTC aims at maximizing the “overall” score of paths in Gctc(θ, T ); for that purpose, it minimizes the Forward score:
CTC(θ, T ) = − logadd π∈Gctc(θ,T ) T∑ t=1 fπt(x) , (2)
where the “logadd” operation, also often called “log-sum-exp” is defined as logadd(a, b) = exp(log(a) + log(b)). This overall score can be efficiently computed with the Forward algorithm. To put things in perspective, if one would replace the logadd(·) by a max(·) in (2) (which can be then efficiently computed by the Viterbi algorithm, the counterpart of the Forward algorithm), one would then maximize the score of the best path, according to the model belief. The logadd(·) can be seen as a smooth version of the max(·): paths with similar scores will be attributed the same weight in the overall score (and hence receive the same gradient), and paths with much larger score will have much more overall weight than paths with low scores. In practice, using the logadd(·) works much better than the max(·). It is also worth noting that maximizing (2) does not diverge, as the acoustic model is assumed to output normalized scores (log-probabilities) fi(·).
In this paper, we explore an alternative to CTC, with three differences: (i) there are no blank labels, (ii) un-normalized scores on the nodes (and possibly un-normalized transition scores on the edges) (iii) global normalization instead of per-frame normalization:
• The advantage of (i) is that it produces a much simpler graph (see Figure 3a and Figure 3b). We found that in practice there was no advantage of having a blank class to model the possible “garbage” frames between letters. Modeling letter repetitions (which is also an important quality of the blank label in CTC) can be easily replaced by repetition character labels (we used two extra labels for two and three repetitions). For example “caterpillar” could be written as “caterpil2ar”, where “2” is a label to represent the repetition of the previous letter. Not having blank labels also simplifies the decoder.
• With (ii) one can easily plug an external language model, which would insert transition scores on the edges of the graph. This could be particularly useful in future work, if one wanted to model representations more high-level than letters. In that respect, avoiding normalized transitions is important to alleviate the problem of “label bias” Bottou (1991); Lafferty et al. (2001). In this work, we limited ourselves to transition scalars, which are learned together with the acoustic model.
• The normalization evoked in (iii) is necessary when using un-normalized scores on nodes or edges; it insures incorrect transcriptions will have a low confidence.
In the following, we name our criterion “Auto Segmentation Criterion” (ASG). Considering the same notations than for CTC in (2), and an unfolded graph Gasg(θ, T ) over T frames for a given transcription θ (as in Figure 3b), as well as a fully connected graph Gfull(θ, T ) over T frames (representing all possible sequence of letters, as in Figure 3c), ASG aims at minimizing:
ASG(θ, T ) = − logadd π∈Gasg(θ,T ) T∑ t=1 (fπt(x) + gπt−1,πt(x)) + logadd π∈Gfull(θ,T ) T∑ t=1 (fπt(x) + gπt−1,πt(x)) , (3) where gi,j(·) is a transition score model to jump from label i to label j. The left-hand part of 3 promotes sequences of letters leading to the right transcription, and the right-hand part demotes all sequences of letters. As for CTC, these two parts can be efficiently computed with the Forward algorithm. Derivatives with respect to fi(·) and gi,j(·) can be obtained (maths are a bit tedious) by applying the chain rule through the Forward recursion.
2.4 BEAM-SEARCH DECODER
We wrote our own one-pass decoder, which performs a simple beam-search with beam threholding, histogram pruning and language model smearing Steinbiss et al. (1994). We kept the decoder as
simple as possible (under 1000 lines of C code). We did not implement any sort of model adaptation before decoding, nor any word graph rescoring. Our decoder relies on KenLM Heafield et al. (2013) for the language modeling part. It also accepts un-normalized acoustic scores (transitions and emissions from the acoustic model) as input. The decoder attempts to maximize the following:
L(θ) = logadd π∈Gasg(θ,T ) T∑ t=1 (fπt(x) + gπt−1,πt(x)) + α logPlm(θ) + β|θ| , (4)
where Plm(θ) is the probability of the language model given a transcription θ, α and β are two hyper-parameters which control the weight of the language model and the word insertion penalty respectively.
3 EXPERIMENTS
3.1 SETUP
We implemented everything using Torch71. The ASG criterion as well as the decoder were implemented in C (and then interfaced into Torch).
We consider as benchmark LibriSpeech, a large speech database freely available for download (Panayotov et al., 2015). LibriSpeech comes with its own train, validation and test sets. Except when specified, we used all the available data (about 1000h of audio files) for training and validating our models. We use the original 16 KHz sampling rate. The vocabulary L contains 30 graphemes: the standard English alphabet plus the apostrophe, silence, and two special “repetition” graphemes which encode the duplication (once or twice) of the previous letter (see Section 2.3).
The architecture hyper-parameters, as well the decoder ones were tuned using the validation set. In the following, we either report letter-error-rates (LERs) or word-error-rates (WERs). WERs have been obtained by using our own decoder (see Section 2.4), with the standard 4-gram language model provided with LibriSpeech2.
1http://www.torch.ch. 2http://www.openslr.org/11.
MFCC features are computed with 13 coefficients, a 25 ms sliding window and 10 ms stride. We included first and second order derivatives. Power spectrum features are computed with a 25 ms window, 10 ms stride, and have 257 components. All features are normalized (mean 0, std 1) per input sequence.
3.2 RESULTS
Table 1 reports a comparison between CTC and ASG, in terms of LER and speed. Our ASG criterion is implemented in C (CPU only), leveraging SSE instructions when possible. Our batching is done with an OpenMP parallel for. We picked the CTC criterion implementation provided by Baidu3. Both criteria lead to the same LER. For comparing the speed, we report performance for sequence sizes as reported initially by Baidu, but also for longer sequence sizes, which corresponds to our average use case. ASG appears faster on long sequences, even though it is running on CPU only. Baidu’s GPU CTC implementation seems more aimed at larger vocabularies (e.g. 5000 Chinese characters).
We also investigated the impact of the training size on the dataset, as well as the effect of a simple data augmentation procedure, where shifts were introduced in the input frames, as well as stretching. For that purpose, we tuned the size of our architectures (given a particular size of the dataset), to avoid over-fitting. Figure 4a shows the augmentation helps for small training set size. However, with enough training data, the effect of data augmentation vanishes, and both type of features appear to perform similarly. Figure 4b reports the WER with respect to the available training data size. We observe that we compare very well against Deep Speech 1 & 2 which were trained with much more data Hannun et al. (2014); Amodei et al. (2015).
Finally, we report in Table 2 the best results of our system so far, trained on 1000h of speech, for each type of features. The overall stride of architectures is 320 (see Figure 1), which produces a label every 20 ms. We found that one could squeeze out about 1% in performance by refining the precision of the output. This is efficiently achieved by shifting the input sequence, and feeding it to the network several times. Results in Table 2 were obtained by a single extra shift of 10 ms. Both power spectrum and raw features are performing slightly worse than MFCCs. One could expect, however, that with enough data (see Figure 4) the gap would vanish.
3https://github.com/baidu-research/warp-ctc.
4 CONCLUSION
We have introduced a simple end-to-end automatic speech recognition system, which combines a standard 1D convolutional neural network, a sequence criterion which can infer the segmentation, and a simple beam-search decoder. The decoding results are competitive on the LibriSpeech corpus with MFCC features (7.2% WER), and promising with power spectrum and raw speech (9.4% WER and 10.1% WER respectively). We showed that our AutoSegCriterion can be faster than CTC (Graves et al., 2006), and as accurate (table 1). Our approach breaks free from HMM/GMM pre-training and force-alignment, as well as not being as computationally intensive as RNN-based approaches (Amodei et al., 2015) (on average, one LibriSpeech sentence is processed in less than 60ms by our ConvNet, and the decoder runs at 8.6x on a single thread).

This submission proposes a letter-level decoder with a variation of the CTC approach they call ASG, where the blank symbol is dropped and replaced by letter repetition symbols, and where explicit normalization is dropped. Both the description of a letter-level model (though not novel), as well as the CTC-variant are interesting.
The approach is evaluated on the LibriSpeech task. The authors claim that their approach is competitive. They compare their modelling variant ASG to CTC, but a comparison of the letter-level approach to available word-level results are missing. Compared to the results obtained in Panayotov et al. 2015, the performance obtained here seems only comparable to word-level GMM/HMM models, but worse than word-level hybrid DNN/HMM models, though Panayotov et al. also appled speaker adaptation, which was not done, as far as I can see. I suggest to add a comparison to Panyotov's results (in addition to mentioning Baidu's results on Librispeech, which are not comparable due to much larger amounts of training data), to allow readers to get a quantitative idea. As pointed out by the authors in the text, Baidu's GPU implementation for CTC is more aimed at larger vocabularies, therefore the comparison to GPU in Tables 1a-c do not seem to be helpful for this work, without further discussing the implementations.
You are using quite a huge analysis window (nearly 2s). Even though other authors also use windows up to 0.5s to 1s (e.g. MRASTA features), some comments on how you arrive at such a large window, and what advantages you observe for it, would be interesting.
The submission is well written, though more details on the experiences with using non-normalized (transition) scores and beam pruning would be desirable. Table 1 would be better readable if the units of the numbers shown in a/b/c would be shown within the tables, and not only in the caption.
Prior (partial) publications of this work (your NIPS end-to-end workshop paper) should clearly be mentioned/referenced.
What do you mean by transition "scalars"?
I do not repeat further comments here, which were already given in the pre-review period.
Minor comments:
 - Sec. 2.3, end of 2nd sentence: train properly the model -> train the model properly
   End of same paragraph: boostrap -> bootstrap (such errors should be avoided by performing an automatic spell check)
 - Sec. 2.3: Bayse -> Bayes
 - definition of logadd is wrong (see comment) - (applies also for your NIPS end-to-end workshop paper).
 - line before Eq. (3): all possible sequence of letters -> all possible sequences of letters (plural)
 - Sec. 2.4, first line: threholding -> thresholding (spell check..)
 - Figure 4: mention the corpus used here - dev?


MS MARCO: A HUMAN GENERATED MACHINE READING COMPREHENSION DATASET
This paper presents our work on the design and development of a new, large scale dataset, which we name MS MARCO, for MAchine Reading COmprehension. This new dataset is aimed to overcome a number of well-known weaknesses of previous publicly available datasets for the same task of reading comprehension and question answering. In MS MARCO, all questions are sampled from real anonymized user queries. The context passages, from which answers in the dataset are derived, are extracted from real web documents using the most advanced version of the Bing search engine. The answers to the queries are human generated. Finally, a subset of these queries has multiple answers. We aim to release one million queries and the corresponding answers in the dataset, which, to the best of our knowledge, is the most comprehensive real-world dataset of its kind in both quantity and quality. We are currently releasing 100,000 queries with their corresponding answers to inspire work in reading comprehension and question answering along with gathering feedback from the research community.
1 INTRODUCTION
Building intelligent agents with the ability for reading comprehension (RC) or open-domain question answering (QA) over real world data is a major goal of artificial intelligence. Such agents can have tremendous value for consumers because they can power personal assistants such as Cortana web (b), Siri web (e), Alexa web (a), or Google Assistant web (d) found on phones or headless devices like Amazon Echo web (c), all of which have been facilitated by recent advances in deep speech recognition technology Hinton et al. (2012); Dahl et al. (2012). As these types of assistants rise in popularity, consumers are finding it more convenient to ask a question and quickly get an answer through voice assistance as opposed to navigating through a search engine result page and web browser. Intelligent agents with RC and QA abilities can also have incredible business value by powering bots that automate customer service agents for business found through messaging or chat interfaces.
Real world RC and QA is an extremely challenging undertaking involving the amalgamation of multiple difficult tasks such as reading, processing, comprehending, inferencing/reasoning, and finally summarizing the answer.
The public availability of large datasets has led to many breakthroughs in AI research. One of the best examples is ImageNet’s Deng et al. (2009) exceptional release of 1.5 million labeled examples and 1000 object categories which has led to better than human level performance on object classification from images He et al. (2015). Another example is the very large speech databases collected over 20 years by DARPA that enabled successes of deep learning in speech recognition Deng & Huang (2004). Recently there has been an influx of datasets for RC and QA as well. These databases, however, all have notable drawbacks. For example, some are not large enough to train deep models Richardson et al. (2013), and others are larger but are synthetic.
One characteristic in most, if not all, of the existing databases for RC and QA research is that the distribution of questions asked in the databases are not from real users. In the creation of most RC or QA datasets, usually crowd workers are asked to create questions for a given piece of text or
document. We have found that the distribution of actual questions users ask intelligent agents can be very different from those conceived from crowdsourcing them from the text.
Furthermore, real-world questions can be messy: they may include typos and abbreviations. Another characteristic of current datasets is that text is often from high-quality stories or content such as Wikipedia. Again, real-world text may have noisy or even conflicting content across multiple documents and our experience is that intelligent agents will often need to operate over this type of problematic data.
Finally, another unrealistic characteristic of current datasets is that answers are often restricted to an entity or a span from the existing reading text. What makes QA difficult in the real world is that an existing entity or a span of text may not be sufficient to answer the question. Finding the best answer as the output of QA systems may require reasoning across multiple pieces of text/passages. Users also prefer answers that can be read in a stand-alone fashion; this sometimes means stitching together information from multiple passages, as the ideal output not only answers the question, but also has supporting information or an explanation.
In this paper we introduce Microsoft MAchine Reading Comprehension (MS MARCO) - a large scale real-world reading comprehension dataset that addresses the shortcomings of the existing datasets for RC and QA discussed above. The questions in the dataset are real anonymized queries issued through Bing or Cortana and the documents are related web pages which may or may not be enough to answer the question. For every question in the dataset, we have asked a crowdsourced worker to answer it, if they can, and to mark relevant passages which provide supporting information for the answer. The answer is strongly encouraged to be in the form of a complete sentence, so the workers may write a longform passage on their own. MS MARCO includes 100,000 questions, 1 million passages, and links to over 200,000 documents. Compared to previous publicly available datasets, this dataset is unique in the sense that (a) all questions are real user queries, (b) the context passages, which answers are derived from, are extracted from real web documents, (c) all the answers to the queries are human generated, (d) a subset of these queries has multiple answers, (e) all queries are tagged with segment information.
2 RELATED WORK
Datasets have played a significant role in making forward progress in difficult domains. The ImageNet dataset Deng et al. (2009) is one of the best known for enabling advances in image classification and detection and inspired new classes of deep learning algorithms Krizhevsky et al. (2012) Girshick et al. (2014) He et al. (2015). Reading comprehension and open domain question answering is one of those domains existing systems still struggle to solve Weston et al. (2015). Here we summarize a couple of the previous approaches towards datasets for reading comprehension and open domain question answering.
One can find a reasonable amount of semi-synthetic reading comprehension and question answering datasets. Since these can be automatically generated they can be large enough to apply modern data intensive models. Hermann et al. created a corpus of cloze style questions from CNN / Daily News summaries Hermann et al. (2015) and Hill et al. has built the Children’s Book Test Hill et al. (2015). Another popular question answering dataset involving reasoning is by Weston et al. Weston et al. (2015). One drawback with these sets is it does not capture the same question characteristics we find with questions people ask in the real world.
MCTest is a challenging dataset which contains 660 stories created by crowdworkers, 4 questions per story, and 4 answer choices per question Richardson et al. (2013), but real-world QA systems needs to go beyond multiple choice answers or selecting from known responses. WikiQA is another set which includes 3047 questions Yang et al. (2015). While other sets are synthetic or editorgenerated questions WikiQA is constructed using a more natural process using actual query logs. It also includes questions for which there are no correct sentences which is an important component in any QA system. Unfortunately, these sets are too small to try data demanding approaches like deep learning.
A more recently introduced reading comprehension dataset is the Stanford Question Answering Dataset (SQuAD) Rajpurkar et al. (2016) which consists of 107785 question/answer pairs from 536 articles where the answer is span of paragraph. A few differences between MS MARCO and SQuAD is (a) SQuAD consisting of questions posed by crowdworkers while MS MARCO is sampled from the real world, (b) SQuAD is on a small set of high quality Wikipedia articles while MS MARCO is from a large set of real web documents, and (c) SQuAD consists of spans while MS MARCO has human generated answers (if there is one).
3 THE MS MARCO DATASET
In order to deliver true machine Reading Comprehension (RC), we start with QA as the initial problem to solve. Our introduction covered some of the key advantages of making very large RC or QA datasets freely available that contain only real-world questions and human crowdsourced answers versus artificially generated data. Given those advantages, our goal is that MS MARCO - a large scale, real-world and human sourced QA dataset - will become a key vehicle to empower researchers to deliver many more AI breakthroughs in the future, just like ImageNet Deng et al. (2009) enabled for image comprehension before.
Additionally, building an RC-oriented dataset helps us understand a contained yet complex RC problem while learning about all of the infrastructure pieces needed to build such a large one-million query set that helps the community make progress on state-of-the-art research problems. This task is also helping us experiment with natural language processing and deep learning models as well as to understand detailed characteristics of the very large training data required to deliver a true AI breakthrough in RC.
This first MS MARCO release contains 100,000 queries with answers to share the rich information and benchmarking capabilities it enables. Our first goal is to inspire the research community to try and solve reading comprehension by building great question answering and related models with the ability to carry out complex reasoning. We also aim to gather feedback and learn from the community towards completing the one-million query dataset in the near future.
This dataset has specific value-added features that distinguish itself from previous datasets freely available to researchers. The following factors describe the uniqueness of the MS MARCO dataset:
• All questions are real, anonymized user queries issued to the Bing search engine. • The context passages, which answers are derived from, are extracted from real Web docu-
ments in the Bing Index. • All of the answers to the queries are human generated. • A subset of these queries has multiple answers. • All queries are tagged with segment information.
The next sections outline the structure, building process and distribution of the MS MARCO dataset along with metrics needed to benchmark answer or passage synthesis and our initial experimentation results.
3.1 DATASET STRUCTURE AND BUILDING PROCESS
The MS MARCO dataset structure is described in Table 2 below.
Starting with the real-world Bing user queries we filter them down to only those that are asking for a question (1) and the Web index documents mentioned in Table 2 as data sources, we automati-
cally extracted context passages from those documents (2). Then, human judges selected relevant passages that helped them write natural language answers to each query in a concise way (3). Following detailed guidelines, judges used a Web-based user interface (UI) to complete this task (3 and 4). A simplified example of such a UI is shown in figure 2.
A feedback cycle and auditing process evaluated dataset quality regularly to ensure answers were accurate and followed the guidelines. In the back-end, we tagged queries with segment classification labels (5) to understand the resulting distribution and the type of data analysis, measurement and experiments this dataset would enable for researchers. Segment tags include
• NUMERIC • ENTITY • LOCATION • PERSON • DESCRIPTION (Phrase)
It is important to note that the question queries above are not artificially handcrafted questions based on Web documents but real user queries issued to Bing over the years. Humans are not always clear, concise or to the point when asking questions to a search engine. An example of a real question query issued to Bing is {in what type of circulation does the oxygenated blood flow between the heart and the cells of the body?}. Unlike previously available datasets, we believe these questions better represent actual human information seeking needs and are more complex to answer compared to artificially generated questions based on a set of documents.
To solve for these types of questions we need a system with human level reading comprehension and reasoning abilities. E.g., given a query such as {will I qualify for osap if i’m new in canada} as shown in figure 2 one of the relevant passages include:
You must be a 1. Canadian citizen, 2. Permanent Resident or 3. Protected person
A RC model needs to parse and understand that being new to a country is usually the opposite of citizen, permanent resident, etc. This is not a simple task to do in a general way. As part of our dataset quality control process, we noticed that even human judges had a hard time reaching this type of conclusions, especially for content belonging to areas they were not familiar with.
The MS MARCO dataset that we are publishing consists of four major components:
• Queries:These are a subset of user queries issued to a commercial search engine wherein the user is looking for a specific answer. This is in contrast to navigational intent which is another major chunk of user queries where the intent is to visit a destination website. The queries were selected through a classifier which was trained towards answer seeking intent of the query based on human labeled data. The query set was further pruned to only contain queries for which the human judges were able to generate an answer based on the passages that were provided to the judges.
• Passages: For each query, we also present a set of approximately 10 passages which might potentially have the answer to the query. These passages are extracted from relevant webpages. The passages were selected through a separate IR (information retrieval) based machine learned system.
• Answers: For each query, the data set also contain one or multiple answers that were generated by human judges. The judge task involved looking at the passages and synthesizing an answer using the content of the passages that best answers the given query.
• Query type: For each query, the dataset also contains the query intent type across five different categories – (a) description, (b) numeric, (c) entity, (d) person and (e) location. For example, ”xbox one release date” will be labeled as numeric while ”how to cook a turkey” will be of type description.This classification is done using a machine learned classifier using human labeled training data. The features of the classifier included unigram/bigram features, brown clustering features, LDA cluster features, dependency parser features, amongst others. The classifier was a multi-class SVM classifier with an accuracy of 90.31% over test data.
Since the query set is coming from real user queries, not all queries explicitly contain ”what”, ”where”, ”how” kind of keywords even though the intents are similar. For example, users could type in a query like ”what is the age of barack obama” as ”barack obama age”. Table 3.1 lists the percentage of queries that explicitly contain the words ”what”, ”where”, etc.
The following table shows the distribution of queries across different answer types as described earlier in this section.
4 EXPERIMENTAL RESULTS
In this section, we present our results over a range of experiments designed to showcase characteristics of MS MARCO dataset. As we discussed in section 3, human judgments are being accumulated in order to grow the dataset to the expected scale. Along the time line various snapshots of the dataset were taken and used in thoughtfully designed experiments for validation and insights. With dataset developing, the finalized experiment results may differ on the complete dataset, however, we expect observations and conclusions to be reasonably representative.
We group the queries in MS MARCO dataset into various categories based on their answer types, as described in subsection 3.1. The complexity of the answers varies greatly from category to category. For example, the answers to Yes/No questions are simply binary. The answers to entity questions can be a single entity name or phrase, such as the answer ”Rome” for query ”What is the capital of Italy”.
However, for other categories such as description queries, a longer textual answer is often required to answer to full extent, such as query ”What is the agenda for Hollande’s state visit to Washington?”. These long textual answers may need to be derived through reasoning across multiple pieces of text. Since we impose no restrictions on the vocabulary used, different human editors often compose for the same query multiple reference answers with different expressions.
Therefore, in our experiments different evaluation metrics are used for different categories many of them presented in an earlier paper Mitra et al.. As shown in subsection 4.1 and 4.2, we use accuracy and precision-recall to measure the quality of the numeric answers, and apply metrics like ROUGE-L Lin (2004) and phrasing-aware evaluation framework Mitra et al. for long textual answers. The phrasing-aware evaluation framework aims to deal with the diversity of natural language in evaluating long textual answers. The evaluation requires a large number of reference answers per question that are each curated by a different human editor, thus providing a natural way to estimate how diversely a group of individuals may phrase the answer to the same question. A family of pairwise similarity based metrics can used to incorporate consensus between different reference answers for evaluation. These metrics are simple modifications to metrics like BLEU Papineni et al. (2002) and METEOR Banerjee & Lavie (2005), and are shown to achieve better correlation with human judgments. Accordingly as part of our experiments, a subset of MS MARCO where each query has multiple answers was used to evaluate model performance with both BLEU and pa-BLEU as metrics.
4.1 GENERATIVE MODEL EXPERIMENTS
Recurrent Neural Networks (RNNs) are capable of predicting future elements from sequence prior. It is often used as a generative language model for various NLP tasks, such as machine translation Bahdanau et al. (2014), query answering Hermann et al. (2015), etc. In this QA experiment setup, we mainly target training and evaluation of such generative models which predict the human-generated answers given queries and/or contextual passages as model input.
Table 5 shows the result quality from these models using ROUGE-L metric. While passages provided in MS MARCO generally contains useful information for given queries, the answer generation
nature of the problem makes it relatively challenging for simple generative models to achieve great results. Model advancement from Seq2Seq to Memory Networks are captured by MS MARCO on ROUGE-L.
Additionally we evaluated Memory Networks model on an MS MARCO subset where queries have multiple answers. Table 6 shows answers quality of the model measured by BLEU and its pairwise variant pa-BLEU.
4.2 CLOZE-STYLE MODEL EXPERIMENTS
Cloze-style test is a representative and fundamental problem in machine reading comprehension. In this test, a model attempts to predict missing symbols in a partially given text sequence by reading context texts that potentially have helpful information. CNN and Daily Mail dataset is one of the most commonly used cloze-style QA dataset. Sizable progress has been made recently from various model proposals in participating cloze-style test competition on these datasets. In this section, we present the performance of two machine reading comprehension models using both CNN test dataset and a MS MARCO subset. The subset is filtered to numeric answer type category, to which clozestyle test is applicable.
We show model accuracy numbers on both datasets in table 7, and precision-recall curves on MS MARCO subset in figure 1.
5 SUMMARY AND FUTURE WORK
The MS MARCO dataset described in this paper above provides training data with question-answer pairs, where only a single answer text is provided via crowdsourcing. This simplicity makes the evaluation relatively easy. However, in the real world, multiple and equally valid answers are possible to a single question. This is akin to machine translation where multiple ways of translation are equally valid. Our immediate future work is to enrich the test set of the current dataset by providing multiple answers. We plan to add 1000 to 5000 such multiple answers in the dateset described in this paper.
Subsequent evaluation experiments on comparing single vs. multiple answers will be conducted to understand whether the model we have built has better resolution with multiple answers. The evaluation metric can be the same METEOR as described in the experiments reported earlier in this paper.
While MS MARCO has overcome a set of undesirable characteristics of the existing RC and QA datasets, notably the requirement that the answers to questions have to be restricted to an entity or a span from the existing reading text. Our longer-term goal is to be able to develop more advanced datasets to assess and facilitate research towards real, human-like reading comprehension. Currently, much of the successes of deep learning has been demonstrated in classification tasks Deng & Yu (2014). Extending this success, the more complex reasoning process in many current deep-learning-based RC and QA methods has relied on multiple stages of memory networks with attention mechanisms and with close supervision information for classification. These artificial memory elements are far away from the human memory mechanism, and they derive their power mainly from the labeled data (single or multiple answers as labels) which guides the learning of network weights using a largely supervised learning paradigm. This is completely different from how human does reasoning. If we ask the current connectionist reasoning models trained on question-answer pairs to do another task such as recommendation or translation that are away from the intended classification task (i.e. answering questions expressed in a pre-fixed vocabulary), they will completely fail. Human cognitive reasoning would not fail in such cases. While recent work is moving towards this important direction Graves et al. (2016), how to develop new deep learning methods towards human-like natural language understanding and reasoning, and how to design more advanced datasets to evaluate and facilitate this research is our longer-term goal.

Summary: The paper proposes a large-scale dataset for reading comprehension, with the final goal of releasing 1 million questions and answers. The authors have currently released 100,000 queries and their answers. The dataset differs from existing reading comprehension datasets mainly w.r.t queries being sampled from user queries rather than being generated by crowd-workers and answers being generated by crowd-workers rather than being spans of text from the provided passage. The paper presents some analysis of the dataset such as distribution of answer types. The paper also presents the results of some generative and some cloze-style models on the MS MARCO dataset.
Strengths:
1. The paper provides useful insights about the limitations of the existing reading comprehension datasets – questions asked by crowd-workers have different distribution compared to that of questions asked by actual users of intelligent agents, answers being restricted to span from the reading text rather than requiring reasoning across multiple pieces of text/passages.
2. MS MARCO dataset has novel useful characteristics compared to existing reading comprehension datasets – questions are sampled from user queries, answers are generated by humans.
3. The experimental evaluation of the existing baseline models on the MS MARCO dataset is satisfactory.
Weaknesses/Suggestions:
1. The paper does not report human performance on the dataset. Human performance should be reported to estimate the difficulty of the dataset. The degree of inter-human agreement will also reflect how well the metric (being used to compute inter-human agreement and accuracies of the baseline models) can deal with variance in the sentence structure with similar semantics.
2. I would like to see the comparison between the answer type distribution in the MS MARCO dataset and that in existing reading comprehension datasets such as SQuAD. This would ground the claim made in the paper the distributions of questions asked by crowd-workers is different from that of user queries.
3. The paper uses automatic metrics such as ROUGE, BLEU for evaluating natural language answers. However, it is known that such metrics poorly correlate with human judgement for tasks such as image caption evaluation (Chen et al., Microsoft COCO Captions: Data Collection and Evaluation Server, CoRR abs/1504.00325 (2015)). So, I wonder how authors justify using such metrics for evaluating open-ended natural language answers.
4. The paper mentions that a classifier was used to filter answer seeking queries from all Bing queries. It would be good to mention the accuracy of this classifier. This will provide insights into what percentage of the MS MARCO questions are answer seeking queries. Similarly, what is the accuracy of the information retrieval based system used to retrieve passages for filtered queries?
5. Please include the description of the best passage baseline in the paper.

6. Fix opening quotes, i.e. ” -> “ (for instance, on page 5, ”what” -> “what”).
Review Summary: The paper is well motivated, the use of user queries and human generated answers makes the dataset different from existing datasets. However, I would like to see the human performance on the dataset and quantitative comparison between the distribution of questions obtained from user queries and that of crowd-sourced questions. I would also like the authors to comment on the use of automatic metrics (such as ROUGE, BLEU) in the light of the fact that such metrics do not correlate well with human judgements for tasks such as image caption evaluation.