It is a surprising fact that electronic medical records are failing at one of their primary purposes, that of tracking the set of medications that the patient is actively taking. Studies estimate that up to 50% of such lists omit active drugs, and that up to 25% of all active medications do not appear on the appropriate patient list. Manual efforts to maintain these lists involve a great deal of tedious human labor, which could be reduced by computational tools to suggest likely missing or incorrect medications on a patient’s list. We report here an application of recurrent neural networks to predict the likely therapeutic classes of medications that a patient is taking, given a sequence of the last 100 billing codes in their record. Our best model was a GRU that achieved high prediction accuracy (micro-averaged AUC 0.93, Label Ranking Loss 0.076), limited by hardware constraints on model size. Additionally, examining individual cases revealed that many of the predictions marked incorrect were likely to be examples of either omitted medications or omitted billing codes, supporting our assertion of a substantial number of errors and omissions in the data, and the likelihood of models such as these to help correct them.
1 INTRODUCTION
The idea of exploiting the large amounts of data captured in electronic medical records for both clinical care and secondary research holds great promise, but its potential is weakened by errors and omissions in those records (Safran et al., 2007; de Lusignan & van Weel, 2006). Among many other problems, accurately capturing the list of medications currently taken by a given patient is extremely challenging (Velo & Minuz, 2009). In one study, over 50% of electronic medication lists contained omissions (Caglar et al., 2011), and in another, 25% of all medications taken by patients were not recorded (Kaboli et al., 2004). Even medication lists provided by the patients themselves contain multiple errors and omissions (Green et al., 2010) .
Many efforts have been made to ensure the correctness of medication lists, most of them involving improved communication between patients and providers (Keogh et al., 2016), but these efforts have not yet been successful, and incorrect or incomplete medication documentation continues to be a source of error in computational medical research. In this work we attempt to identify likely errors and omissions in the record, predicting the set of active medications from the sequence of most recent disease-based billing codes in the record. Predictions from such a model could be used either in manual medication reconciliation (a common process undertaken to correct the medication record) or to provide a prior to other models, such as an NLP model attempting to extract medication use from the narrative clinical text.
Given the sequential nature of clinical data, we suspected that recurrent neural networks would be a good architecture for making these predictions. In this work we investigate this potential, comparing the performance of recurrent networks to that of similarly-configured feed forward networks.
The input for each case is a sequence of ICD-9 billing codes (Section 2.1), for which the model produces a single, multi-label prediction of the therapeutic classes (Section 3.1) of medications taken by the patient during the period of time covered by the billing code sequence.
This work is designed to test how well the complete set of medications a patient is actively taking at a given moment can be predicted by the sequence of diagnostic billing codes leading up to that moment, in the context of non-trivial label noise. It also explores whether sequence-oriented recursive neural nets can do a better job of that prediction than standard feed-forward networks.
2 BACKGROUND
2.1 MEDICAL BILLING CODES
Each time a patient has billable contact with the healthcare system, one or more date-stamped billing codes are attached to the patient record, indicating the medical conditions that are associated (or suspected to be associated) with the reason for the visit. While these codes are notoriously unreliable because they are only used for billing and not actual clinical practice (O’Malley et al., 2005), they are nevertheless useful in a research context (Bastarache & Denny, 2011; Denny et al., 2010), especially if they are used probabilistically (Lasko, 2014). In our institution, codes from the International Classification of Diseases, Ninth Revision (ICD-9) have historically been used, although we have recently transitioned to the tenth revision (ICD-10). For this project, we used ICD-9 codes.
The ICD-9 hierarchy consists of 21 chapters roughly corresponding to a single organ system or pathologic class (Appendix B). Leaf-level codes in that tree represent single diseases or disease subtypes. For this project, we used a subset of the two thousand most common leaf-level codes as our input data.
2.2 RECURRENT NEURAL NETWORKS AND VARIATIONS
Most of the ICLR community are very familiar with recurrent neural networks and their variations, but we include a conceptual description of them here for readers coming from other fields. More thorough descriptions are available elsewhere (Graves, 2012; Olah, 2015).
A recurrent neural network is a variation in which the output of one node on input xt loops around to become an input to another node on input xt+1, allowing information to be preserved as it iterates over an input data sequence (Figure 1). They were introduced in the 1980s (Rumelhart et al., 1986), but achieved explosive popularity only recently, after the development of methods to more reliably capture long-term dependencies, which significantly improved their performance on sequence-tosequence mapping (Hochreiter & Schmidhuber, 1997; Sutskever et al., 2014).
The basic RNN unit has a simple internal structure (Figure 2a). Output from the previous iteration ht−1 and the next input in a sequence xt are both fed to the network on the next iteration. The Long Short-Term Memory configuration (LSTM) introduces new, more complex internal structure (Figure 2b) consisting of four neural network layers and a cell state (ct), which is carried from one iteration to another. The additional layers form forget, input and output gates, which allow for the information to be forgotten (reset) or passed on to varying degrees.
The LSTM model and its variations are commonly used in applications where sequence and temporal data are involved, such as in image captioning (Vinyals et al., 2014), language translation (Sutskever et al., 2014), and speech recognition (Graves et al., 2013). In many cases LSTM models define the state of the art, such as with a recent conversational speech recognizer that (slightly) outperforms professional transcriptionists (Xiong et al., 2016).
A recent variation on the LSTM architecture is the Gated Recurrent Unit (GRU) (Cho et al., 2014), which introduces a single update gate in place of input and forget gates (Figure 2c). GRUs perform as well as or better than LSTMs in many cases (Chung et al., 2014; Jozefowicz et al., 2015), and have the additional advantage of a simpler structure.
In this work we try both an LSTM and a GRU on our learning problem.
2.3 RELATED WORK
Little research in the computational medical domain has used recurrent neural networks. The earliest example we are aware of is the use of an LSTM model that produced reasonable accuracy
(micro-AUC 0.86) in a 128-dimensional multi-label prediction of diagnoses from regularly sampled, continuously-monitored, real-valued physiologic variables in an Intensive Care Unit setting. This was an interesting initial application, but it turned out to be only 0.001 better than the baseline classifier, which was a multi-layer perceptron with expert-designed features (Lipton et al., 2016). Given the dataset size (10,401 patient records) the lack of improvement may have been due to insufficient data to power accurate feature learning in the recurrent network.
Very recent work, contemporary with ours, used a GRU model with a semantic embedding in 32,787 patient records to predict the development of heart failure 3 - 6 months in the future, from medication orders and billing codes in an 18-month window. The model achieved respectable accuracy (0.88 AUC), and demonstrated a meaningful 0.05 AUC improvement over a deep feedforward network (Choi et al., 2016b).
Other recent work from the same group used a GRU model in a multi-label context to predict the medications, billing codes, and time of the next patient visit from a sequence of that same information for previous visits, using 263,706 patient records. It achieved a recall@30 of 72.4 for the task, an improvement of 20 over a single-hidden-layer MLP with 2000 units (Choi et al., 2016a). This is an example of using one of the strengths of a recurrent network - predicting the next element in a sequence. It contrasts with our work that exploits a different strength of recurrent networks - predicting a sequence or class that is semantically distinct from but parallel to the elements of the input sequence.
The closest work to ours from a medical domain perspective is a series of collaborative filter models (including co-occurrence counting, k-nearest neighbors, and logistic regression) that predict missing medications using a leave-one-drug-out evaluation design, with predictions based on the rest of the medications, ICD-9 billing codes, and demographic data. The models were trained and tested on data from 419 patients in three different clinics, with accuracy varying by clinic, as expected, but not appreciably by model. Most models ranked the missing drug in the top 10 results between 40 and 50% of the time, and ranked the therapeutic class of the drug in the top 10 results between 50 and 65% of the time.
Many aspects of our work can be found in these prior efforts, but none addresses our particular problem in the same way. Our work is unique in its learning problem of identifying all drugs a patient is likely to be taking, based only on the billing codes in the record. Like most others cited, we use recurrent neural networks in a multi-label predictive context, but in contrast to them we compare
to the most similar non-recurrent model we can construct, in order to evaluate the contribution of the temporal sequence information to the solution. Finally, we use one to four orders of magnitude more data (3.3 million instances, see Section 3.1) than these prior efforts, which we hope will give us a more realistic assessment of the various deep architectures we use on our problem.
3 EXPERIMENTS
3.1 DATA
Our source database was the deidentified mirror of Vanderbilt’s Electronic Medical Record, which contains billing codes, medication histories, laboratory test results, narrative text and medical imaging data for over 2 million patients, reaching back nearly 30 years (Roden et al., 2008). We obtained IRB approval to use this data in this research.
For this experiment we filtered all records in our database to include only the top 1,000 most common medications and the top m = 2000 most common billing codes, which cover 99.5% of all medication occurrences and 85.1% of all billing code occurrences. We then included all records from the filtered data that had at least one medication occurrence and at least ten billing code occurrences. This resulted in 610,076 complete patient records, which we divided 80/5/15 into training, validation, and final test sets.
A data instance d = {E, T, y} consisted of a sequence E = {e1, . . . , en}, of one-hot billing code vectors ei ∈ {0, 1}m and their associated times T = {t1, . . . , tn}, ti ∈ R as input, and a multi-label vector y ∈ {0, 1}k of medication classes as the output target. The most recent n = 100 billing codes to a selected reference time point in a given patient record were collected into the input sequence E, and their occurrence times into T , zero padding if necessary. All medications that occurred during the time span of T were then collected into the output vector y. Practice patterns change over time, so simply taking the most recent 100 codes for each patient could produce a biased result. To avoid this, we chose random reference points, stratified by medication. In other words, the reference points were randomly chosen from the occurrences of each medication in the entire dataset, up to 10,000 points per medication. This resulted in 3.3 million data instances, an average of 5.4 instances per patient record. Each patient’s data was included in at most one of the training, validation, or test sets.
Because there are often many approximately equivalent medication choices for a given therapeutic purpose, we converted medication names to their therapeutic class (beta blocker, immunosuppressant, corticosteroid, etc.) as a synonym reduction step. This step also aggregated generic with brand names, as well as different formulations of the same active ingredient. For this task we used the Anatomical Chemical Classification System (ATC)1, which is a multi-level ontology of medications, organized by both anatomic and therapeutic class. The top level is a broad categorization of medications (Appendix B), the bottom (fifth) level is individual medications, and we used the third level, which contains 287 therapeutic classes of the approximately appropriate abstraction level for our purpose. We used a publicly available mapping2 to translate between our medication names and ATC codes, with manual mapping for the minority of medications that had no mapping entry. Our set of medications used k = 182 third-level ATC codes, rendering our output label a 182-elementlong multi-label vector, in which an element is set yi = 1 if a medication in that class appeared in the set of medications identified for that instance, yi = 0 otherwise. Some medications mapped to more than one class, and we set yi = 1 for all of them.
Our medication data was collected from structured order entry records and extracted using NLP (Xu et al., 2010) from mentions in the narrative text of a patient record that included the medication name, dose, route and frequency. As discussed above, we assumed (and our results demonstrate) that the medication data is incomplete, and our hope was that a model learned from a sufficiently large dataset will be robust to the missing data.
This configuration represents the input billing codes in a sequence, but the output medications as a multi-label vector. This is because ICD-9 codes are represented sequentially in our source data, but medications are not. They are represented as a list that changes over time in the record. The
1http://www.whocc.no/atc/structure and principles/ 2https://www.nlm.nih.gov/research/umls/rxnorm/
usual goal of clinicians is to verify the list of medications at each visit, and if omissions or additions are indicated by the patient, to change the list to reflect that. But in the time-constrained reality of clinical practice, this reconciliation happens sporadically, and many clinicians are hesitant to change an entry on the medication list for which they were not the original prescriber, so the timing of the changes in the documentation do not reflect the timing of changes in reality. Therefore we are reduced to predicting a single multi-label vector, representing the medications that the patient probably took during the span of time represented by the input codes. (We actually did attempt some full sequence-to-sequence mappings, with various orderings of the medication sequences, but we did not achieve any promising results in that direction.)
3.2 CLASSIFIERS
Our main technical goal was to test the performance of recurrent neural networks on this sequencecentric prediction problem. To evaluate the specific gains provided by the recurrent architectures, we compare performance against a fully connected feed-forward network configured as similarly as possible to the recurrent networks, and (as baselines) a random forest and a constant-prevalence model. We discuss the specific configurations of these classifiers in this section.
3.2.1 RECURRENT NEURAL NETWORKS
We tested both LSTMs and GRUs in this experiment. We configured both architectures to first compute a semantic embedding xi ∈ Rb of each input ei vector, before appending the times ti (Figure 3) and feeding the result to three layers of recurrent units. The final output from the last pass of recurrent unit is as a multi-label prediction for each candidate medication.
The optimal hyperparameters for the model were selected in the randomized parameter optimization (Bergstra & Bengio, 2012), with the embedding dimension b = 32, number of layers, and number of nodes optimized by a few trials of human-guided search. Other optimized parameters included the fraction of dropout (between layers, input gates and recurrent connections), and L1 and L2 regularization coefficients (final values are presented in Appendix A).
Both models were implemented using Keras (Chollet, 2015) and trained for 300 iterations using cross-entropy under the Adadelta optimizer (Zeiler, 2012).
3.2.2 FULLY CONNECTED NEURAL NETWORK
The fully connected network used as similar an architecture as possible to the recurrent networks, in an attempt to isolate the gain achieved from the recurrence property. Specifically, we used the same architecture for embedding and timestamp appending (Figure 3).
Hyperparameters were optimized using random search over the number of layers, number of nodes, dropout, activation function between layers, L1 and L2 regularization coefficients (Appendix A). (Surprisingly, the optimizer chose tanh over ReLU as the optimal activation function.)
The models were also implemented using Keras, and were trained using cross-entropy for 500 iterations under the Adadelta optimizer.
3.2.3 RANDOM FOREST
Because the random forest model is not easily structured to operate on sequences, we represented the input data as either binary occurrence vectors v ∈ {0, 1}m, or bag-of-codes vectors w ∈ Nm (counts of each code value in the sequence) rather than as sequences of codes with associated times. No embedding was used, because random forest code was not able to cope with the large size of the data in the (dense) embedded space.
Even in the (sparse) original space, the full dataset was too large for the random forest code, so we implemented it as an ensemble of ten independent forests, each trained on one tenth of the training data, and their average score used for test predictions.
Models were implemented using scikit-learn (Pedregosa et al., 2011) with parameters optimized under random search (Appendix A).
While other models could reasonably serve as a baseline for this work, we chose a random forest because they tend to perform well on widely varying datasets (Fernández-Delgado et al., 2014), they are efficient to train and test, and they don’t require a huge effort to optimize (in order to produce a fair comparison).
3.3 CONSTANT-PREVALENCE MODEL
This minimum baseline model simply predicts the prevalence of each label for all instances. For example, if there were three possible medications, with prevalences of 0.3, 0.9, and 0.2, then the prediction of this model would be a constant [0.3, 0.9, 0.2] for each instance. We include this model in order to mitigate the fact that while all of our evaluation measures are suitable for comparing models on the same data, some are not well suited for external comparison because they depend, for example, on the prevalence of positive labels (Section 3.4). By including this model we can at least establish a true minimum baseline for reference.
3.4 EVALUATION
Our main evaluation focused on the models, although we also performed a separate evaluation of the embedding.
3.4.1 MODELS
There are several possibilities for evaluation in a multi-label classification context (Sechidis et al., 2011; Zhang & Zhou, 2014). We chose micro-averaged area under the ROC curve (AUC) and label ranking loss as the primary methods of evaluation, because they treat each instance with equal weight, regardless of the nature of the positive labels for that instance. In other words, we wanted primary measures that did not give a scoring advantage to instances with either very many or very few positive labels, or that included very rare or very prevalent labels. Additionally, both of these measures appeal to us as intuitive extensions of the usual binary AUC, when seen from the perspective of a single instance. However, because these two measures don’t reflect all aspects of multi-label prediction performance, we also include macro-averaged AUC, label ranking average precision and coverage error measures.
Micro-averaged AUC considers each of the multiple label predictions in each instance as either true or false, and then computes the binary AUC as if they all belonged to the same 2-class problem (Zhang & Zhou, 2014). In other words, micro-averaged AUC Aµ is:
Aµ = ∣∣(x, x′, l, l′) : f(x, l) ≥ f(x′, l′), (x, l),∈ S, (x′, l′) ∈ S̄∣∣∣∣S∣∣∣∣S̄∣∣ , (1) where S = {(x, l) : l ∈ Y } is the set of (instance, label) pairs with a positive label, and Y = {yi : yi = 1, i = 1 . . . k} is the set of positive labels for input x. Label ranking loss LR gives the average fraction of all possible (positive, negative) label pairs for each instance in which the negative label has a higher score than the positive label (Tsoumakas et al., 2010):
LR = 1
N N∑ j=1
1
|Y (j)||Y (j)| ∣∣∣{(l, l′) : r(j)(l) > r(j)(l′), (l, l′) ∈ Y (j) × Y (j) }∣∣∣ (2) where the superscript (j) refers to the jth test instance (of N total instances) and r(l) is the predicted rank of a label l.
Macro-averaged AUC can be thought of as averaging the AUC performance of several one-vs-all classifiers, one model for each label. It treats each model equally, regardless of the prevalence of positive labels for that model. This gives a score of 0.5 to the constant-prevalence model, at the cost of weighting instances differently in order to achieve that. This is in contrast to micro-averaged AUC, which can be thought of as averaging across instances rather than labels. It weighs each instance equally, at the cost of a 0.5 score no longer being the random-guessing baseline.
Label ranking average precision gives the mean fraction of correct positive labels among all positive labels with lower scores for each label. The coverage error function calculates the mean number of labels on the ranked list that are needed to cover all the positive labels of the sample. Both of these depend on the prevalence of positive labels in a test instance.
3.4.2 EMBEDDING
We evaluated the embedding based on how strongly related in a clinical semantic sense the nearest neighbor to each code is (in the embedding space). A licensed physician manually annotated the list of all 2000 codes with its match category m ∈ {strongly related,loosely related,unrelated}, and we computed the empirical marginal probability P (m) of each category, the empirical conditional probability P (m|d) of the match category given the nearest neighbor (Manhattan) distance d and the empirical marginal probability P (d). For comparison, we computed P (m) under 100 random code pairings.
4 RESULTS AND DISCUSSION
The GRU model had the top performance by all measures, although the LSTM was a close second (Table 1), a performance pattern consistent with previous reports (Chung et al., 2014). The deep neural net performance was about 0.01 worse in both measures, suggesting that the recurrent models were able to use the sequence information, but only to a small advantage over the most similar nontemporal architecture. However, we note that both RNNs’ performance peaked at the top end of our tractable range for model size, while the feed-forward network peaked using a model about one third that size (Appendix A). Experimenting with the architecture, we found that increasing the number of nodes or layers for the feed-forward network increased training time but not performance. This suggests that the RNN performance was limited by the hardware available, and increasing the size of the model may further increase performance, and that the feed-forward network was limited by something else.
Both random forest models were weaker than the deep neural net, as might be expected from the need to resort to binary and bag-of-codes representations of the input data.
A natural question is what performance is good enough for clinical use. While there is little clinical experience with multi-label classifiers, we would generally expect clinicians using a binary classifier in an advisory role to find an AUC & 0.9 to be useful, and AUC & 0.95 to be very useful. An AUC difference of 0.01, and perhaps 0.005 are potentially noticeable in clinical use.
This 0.9/0.01 rule of thumb may loosely translate to our AUC variants, but it can directly translate to Label Ranking Loss LR (2). If we think of a single output prediction ŷ ∈ [0, 1]k as a set of predictions for k binary labels, then 1− AUC for that set of predictions is equivalent to LR for the original instance ŷ. Therefore, values of LR . 0.1 may be clinically useful, and LR . 0.05 may be very useful.
Subjectively examining performance on 20 randomly selected cases, we find very good detailed predictions, but also evidence of both missing medications and missing billing codes. An example of a good set of detailed predictions is from a complex patient suffering from multiple myeloma (a type of cancer) with various complications. This patient was taking 26 medications, 24 of which had moderate to high probability predictions (Figure 4). (We have found by eyeball that a prediction cutoff of 0.2 gives a reasonable balance between sensitivity and specificity for our model.) In the other direction, only two of the high-prediction classes were not actually being taken, but those classes, along with several of the other moderately-predicted classes, are commonly used for cancer and are clinically reasonable for the case. (Details of this and the two cases below are in Appendix C).
A good example of missing medications is a case in which the record has multiple billing codes for both osteoporosis (which is very commonly treated with medication) and postablative hypothyroidism (a deliberately induced condition that is always treated with medication), but no medications of the appropriate classes were in the record. The GRU model predicted both of these classes, which the patient was almost surely taking.
A good example of either missing billing codes or discontinued medications that remain documented as active is a case in which the record has at least five years of data consisting only of codes for Parkinson’s disease, but which lists medications for high cholesterol, hypertension, and other heart disease. The GRU model predicted a reasonable set of medications for Parkinson’s disease and its complications, but did not predict the other medications that are not suggested by the record.
Given how easy it was to find cases with apparently missing codes and medications, we conclude that there is indeed a substantial amount of label noise in our data, and we therefore interpret our models’ performance as lower bounds on the actual performance. We are encouraged that this kind of a model may actually be useful for identifying missing medications in the record, but of course a more thorough validation, and possibly a more accurate model, would be necessary before using in a clinical scenario. A definitive experiment would use off-line research, including reconciling information from various electronic and human sources to establish the ground truth of which medications were being taken on a particular day, but such efforts are labor intensive and expensive, and can only be conducted on a very small scale.
An interesting byproduct of these models is the semantic embedding of ICD-9 codes used in the recurrent networks (Figure 5). Transforming input to a semantic embedding is a common pre-
processing step to improve performance, but clearly the semantic understanding it provides to an algorithm can be useful beyond the immediate learning problem (Mikolov et al., 2013). Investigating the embedding learned in this experiment shows some generalizable potential, but it also reveals the need for further refinement before it can be truly useful. Specifically, while it’s easy to find tight groups of ICD-9 codes that are strongly clinically related in our embedding, we also find groups for which we cannot see a meaningful clinical relationship.
For example, we see two groups of codes relating to kidney failure and diabetes mellitus, two classes of very prevalent disease (Figure 5, insets). In other iterations with different parameter settings, the kidney failure codes were even embedded in a sequence reflecting the natural progression of the disease, with the code for dialysis (an intensive treatment for end-stage kidney failure) embedded at the appropriate place. Interestingly, these were not the parameter settings that optimized overall prediction performance. In other settings, such as our performance-optimal setting, the sequence is close to the natural progression of the disease, but not quite identical. Nevertheless, this is an exciting result that suggests great potential.
Further evaluation of the embedding found that 49% of codes were strongly related semantically to their nearest neighbor, 10% were loosely related, and 41% unrelated. This fraction of strongly related nearest neighbors was lower than we had hoped, but much higher than expected by chance (Figure 6), and it definitely improved classification performance. Furthermore, it was obvious by inspection that in general, codes closer in the embedding were more semantically related than distant codes, but interestingly, the distance to the nearest such neighbor showed the opposite relationship — nearest neighbors that were very close were less likely to be semantically related than nearest neighbors that were far, and this trend is roughly linear across the full range of d (Figure 6). So the sparser the points are in the embedded space, the more semantically related they are to their nearest neighbor, but the causal direction of that effect and the technical reason for it are beyond the scope of this initial work.
For this prediction problem, we settled on predicting the medications that occurred in the record during the same time span as the billing codes used. Originally, we intended to predict only the medications listed on the day of the reference point, but that turned out to greatly exacerbate the missing medication problem. After trying medications that fell on the reference day only, the week prior to the reference day, and the six months prior, our best performance both subjectively and objectively was achieved using the full time range of the input data.
While the performance of the recurrent networks was quite good, we believe it could be improved by including additional input data, such as laboratory test results, demographics, and perhaps vital
signs. We also suspect that if we can devise a way to convert our medication data into reliablyordered sequences, we can more fully exploit the strengths of recurrent networks for medication prediction. We look forward to trying these and other variations in future work.
ACKNOWLEDGMENTS
This work was funded by grants from the Edward Mallinckrodt, Jr. Foundation and the National Institutes of Health R21LM011664 and R01EB020666. Clinical data was provided by the Vanderbilt Synthetic Derivative, which is supported by institutional funding and by the Vanderbilt CTSA grant ULTR000445.
APPENDIX A.
This appendix lists the optimized parameters for the different models. Except where noted, parameters were optimized under random search.
Recurrent Neural Network Models: (parameters marked with an asterisk were optimized with human-guided search.)
Parameter Model
GRU LSTM
Dropout for input gates 0.1 0.25 Dropout for recurrent connections 0.75 0.75 L1 applied to the input weights matrices 0 0 L1 applied to the recurrent weights matrices 0 0 L2 applied to the input weights matrices 0.0001 0.0001 L2 applied to the recurrent weights matrices 0.0001 0.001 L2 applied to the output layer’s weights matrices 0.0001 0.001 Dropout before the output layer 0.5 0.5 *Number of recurrent layers 3 3 *Number of nodes in recurrent units 400 400
Feed Forward Neural Network Model:
Parameter Value
Dropout before the output layer 0.1 Dropout between feed-forward layers 0.1 Number of feed-forward layers 3 Activation function between feed-forward layers tanh Number of nodes in feed-forward layers 128
Random Forest Model (binary input):
Parameter Value
Number of estimators 800 Ratio of features to consider when looking for the best split 0.4666 Minimum number of samples required to split an internal node 87 Minimum number of samples required to be at a leaf node 3 The function to measure the quality of a split entropy
APPENDIX B.
This appendix lists the top level classes for International Statistical Classification of Diseases and Related Health Problems, Ninth Revision (ICD-9) and Anatomical Chemical Classification System (ATC).
ICD-9 chapters.
Code range Description
001-139 Infectious and parasitic diseases 140-239 Neoplasms 240-279 Endocrine, nutritional and metabolic diseases, and immunity disorders 280-289 Diseases of the blood and blood-forming organs 290-319 Mental disorders 320-359 Diseases of the nervous system 360-389 Diseases of the sense organs 390-459 Diseases of the circulatory system 460-519 Diseases of the respiratory system 520-579 Diseases of the digestive system 580-629 Diseases of the genitourinary system 630-679 Complications of pregnancy, childbirth, and the puerperium 680-709 Diseases of the skin and subcutaneous tissue 710-739 Diseases of the musculoskeletal system and connective tissue 740-759 Congenital anomalies 760-779 Certain conditions originating in the perinatal period 780-799 Symptoms, signs, and ill-defined conditions 800-999 Injury and poisoning
V01-V91 Supplementary - factors influencing health status and contact with health services E000-E999 Supplementary - external causes of injury and poisoning
Top level groups ATC codes and their corresponding colors used in Figure 4 and Appendix C.
Code Contents Color
A Alimentary tract and metabolism B Blood and blood forming organs C Cardiovascular system D Dermatologicals G Genito-urinary system and sex hormones H Systemic hormonal preparations, excluding sex hormones and insulins J Antiinfectives for systemic use L Antineoplastic and immunomodulating agents M Musculo-skeletal system N Nervous system P Antiparasitic products, insecticides and repellents R Respiratory system S Sensory organs V Various
APPENDIX C.
This appendix presents results from three illustrative cases from the dozen cases randomly selected for individual evaluation.
CASE 1.
ICD-9 code Code description Time estimate (ago)
203.00 Multiple myeloma, without mention of having achieved remission 4.8 months ago 273.1 Monoclonal paraproteinemia 4.8 months ago 285.9 Anemia, unspecified 4.8 months ago 276.50 Volume depletion, unspecified 4.8 months ago 733.00 Osteoporosis, unspecified 4.8 months ago 203.00 Multiple myeloma, without mention of having achieved remission 4.8 months ago 203.00 Multiple myeloma, without mention of having achieved remission 2.9 months ago 203.01 Multiple myeloma, in remission 2.9 months ago 273.1 Monoclonal paraproteinemia 2.9 months ago 273.1 Monoclonal paraproteinemia 1.6 months ago 279.3 Unspecified immunity deficiency 1.6 months ago 203.00 Multiple myeloma, without mention of having achieved remission 1.6 months ago 781.2 Abnormality of gait 3.7 weeks ago 203.00 Multiple myeloma, without mention of having achieved remission 3.7 weeks ago 401.9 Unspecified essential hypertension 3.7 weeks ago V12.54 Personal history of transient ischemic attack (TIA), and cerebral infarction without residual deficits 3.7 weeks ago 794.31 Nonspecific abnormal electrocardiogram [ECG] [EKG] 3.7 weeks ago 786.09 Other respiratory abnormalities 3.7 weeks ago 273.1 Monoclonal paraproteinemia 3.7 weeks ago 203.00 Multiple myeloma, without mention of having achieved remission 3.6 weeks ago V58.69 Long-term (current) use of other medications 3.6 weeks ago 794.31 Nonspecific abnormal electrocardiogram [ECG] [EKG] 3.4 weeks ago 203.00 Multiple myeloma, without mention of having achieved remission 4 days ago V42.82 Peripheral stem cells replaced by transplant 4 days ago 203.01 Multiple myeloma, in remission 3 days ago
38.97 Central venous catheter placement with guidance 3 days ago V42.82 Peripheral stem cells replaced by transplant 3 days ago V58.81 Fitting and adjustment of vascular catheter 3 days ago 203.00 Multiple myeloma, without mention of having achieved remission 3 days ago V42.82 Peripheral stem cells replaced by transplant 2 days ago 203.01 Multiple myeloma, in remission 2 days ago 203.00 Multiple myeloma, without mention of having achieved remission 1 day ago V42.82 Peripheral stem cells replaced by transplant 1 day ago 203.00 Multiple myeloma, without mention of having achieved remission now V42.82 Peripheral stem cells replaced by transplant now
Medication predictions for a complicated patient. Each vertical bar represents the prediction for a single medication class, with the height of the bar representing the confidence of the prediction. Black labels above arrows indicate ATC therapeutic classes for medications the patient was actually taking. Colors and letters below the axis indicate high-level therapeutic class groups.
Predicted vs. actual medication classes for the patient in Case 1. The four-character sequence in the first and fourth columns is the ATC code for the medication therapeutic class, and an asterisk in the first column indicates that the predicted medication is in the actual medication list. Probabilities listed are the model predictions for the listed therapeutic class. In the predicted medications column, all predictions with probability at least 0.2 are listed.
Top predictions Prob. True labels Prob.
S03B* Corticosteroids 97.01% S03B Corticosteroids 97.01% S01C* Antiinflammatory agents and antiinfectives in combi-
nation 95.54% S01C Antiinflammatory agents and antiinfectives in combination 95.54%
S02B* Corticosteroids 95.54% S02B Corticosteroids 95.54% L01A Alkylating agents 94.00% D07X Corticosteroids, other combinations 93.37% D07X* Corticosteroids, other combinations 93.37% H02A Corticosteroids for systemic use, plain 91.06% H02A* Corticosteroids for systemic use, plain 91.06% D07A Corticosteroids, plain 90.83% D07A* Corticosteroids, plain 90.83% S01B Antiinflammatory agents 90.79% S01B* Antiinflammatory agents 90.79% D10A Anti-acne preparations for topical use 88.56% D10A* Anti-acne preparations for topical use 88.56% C05A Agents for treatment of hemorrhoids and anal fissures
for topical use 88.52%
C05A* Agents for treatment of hemorrhoids and anal fissures for topical use 88.52% R01A Decongestants and other nasal preparations for topical use 87.02% A04A Antiemetics and antinauseants 87.95% J05A Direct acting antivirals 86.83% R01A* Decongestants and other nasal preparations for topi-
cal use 87.02% A01A Stomatological preparations 86.11%
J05A* Direct acting antivirals 86.8% N02A Opioids 84.86% A01A* Stomatological preparations 86.11% B05C Irrigating solutions 82.56% N02A* Opioids 84.86% A12C Other mineral supplements 79.50% B05C* Irrigating solutions 82.56% B05X I.V. solution additives 74.84% A12C* Other mineral supplements 79.50% L04A Immunosuppressants 68.76% B05X* I.v. solution additives 74.84% N02B Other analgesics and antipyretics 57.24% L04A* Immunosuppressants 68.76% S01A Antiinfectives 54.59% N05A Antipsychotics 58.64% J01D Other beta-lactam antibacterials 43.40% N02B* Other analgesics and antipyretics 57.24% C03C High-ceiling diuretics 39.88% S01A* Antiinfectives 54.59% J01M Quinolone antibacterials 29.78% L03A Immunostimulants 45.96% C07A Beta blocking agents 27.08% A02B Drugs for peptic ulcer and gastro-oesophageal reflux
disease 44.56%
J01D* Other beta-lactam antibacterials 43.40% N03A Antiepileptics 20.00% C03C* High-ceiling diuretics 39.88% J01X Other antibacterials 5.88% B01A Antithrombotic agents 37.80% M03B Muscle relaxants, centrally acting agents 5.09% V03A All other therapeutic products 34.18% R06A Antihistamines for systemic use 31.78% A06A Drugs for constipation 31.57% J01M* Quinolone antibacterials 29.78% N05B Anxiolytics 29.42% D04A Antipruritics, incl. antihistamines, anesthetics, etc. 27.62% C07A* Beta blocking agents 27.08% L01X Other antineoplastic agents 24.72% R05C Expectorants, excl. combinations with cough sup-
pressants 20.43%
N03A* Antiepileptics 20.00%
CASE 2.
ICD-9 code Code description Time estimate (ago)
735.4 Other hammer toe (acquired) 2.4 years ago 729.5 Pain in limb 2.4 years ago 244.1 Other postablative hypothyroidism 1.5 years ago 285.9 Anemia, unspecified 1.5 years ago 244.1 Other postablative hypothyroidism 1.2 years ago 244.1 Other postablative hypothyroidism 11.5 months ago
733.00 Osteoporosis, unspecified 11.5 months ago 733.01 Senile osteoporosis 7.7 months ago 268.9 Unspecified vitamin D deficiency 7.7 months ago 729.5 Pain in limb 7.7 months ago 174.9 Malignant neoplasm of breast (female), unspecified 7.7 months ago 722.52 Degeneration of lumbar or lumbosacral intervertebral disc 7.7 months ago 279.3 Unspecified immunity deficiency 7.7 months ago 733.01 Senile osteoporosis 6.4 months ago 733.01 Senile osteoporosis 6.2 months ago 244.1 Other postablative hypothyroidism 6.0 months ago 401.1 Benign essential hypertension 6.0 months ago V58.69 Long-term (current) use of other medications 1.9 weeks ago 733.01 Senile osteoporosis now 244.1 Other postablative hypothyroidism now V58.69 Long-term (current) use of other medications now
Predicted vs. actual medication classes for Case 2. Table structure as in case 1.
Top predictions Prob. True labels Prob.
M05B Drugs affecting bone structure and mineralization 88.18% A11C Vitamin a and d, incl. combinations of the two 39.42% H03A Thyroid preparations 84.82% N06A Antidepressants 20.88% H05A Parathyroid hormones and analogues 66.33% C10A Lipid modifying agents, plain 17.05% A11C* Vitamin a and d, incl. combinations of the two 39.42% N03A Antiepileptics 15.61% N02B Other analgesics and antipyretics 37.58% C09C Angiotensin ii antagonists, plain 10.38% A01A Stomatological preparations 23.05% L02B Hormone antagonists and related agents 4.22% A12A Calcium 21.59% N06A* Antidepressants 20.88% C07A Beta blocking agents 20.81%
Medication predictions for a simpler patient. Note that the high-prediction medications are clinically reasonable given the billing codes in the sequence. Figure representation as in case 1.
CASE 3.
ICD-9 code Code description Time estimate (ago)
332.0 Paralysis agitans 5.0 years ago 332.0 Paralysis agitans 4.7 years ago 332.0 Paralysis agitans 4.5 years ago 332.0 Paralysis agitans 4.0 years ago 332.0 Paralysis agitans 3.5 years ago 332.0 Paralysis agitans 3.0 years ago 332.0 Paralysis agitans 2.7 years ago 332.0 Paralysis agitans 2.4 years ago 332.0 Paralysis agitans 2.0 years ago 332.0 Paralysis agitans 1.7 years ago 332.0 Paralysis agitans 1.0 years ago 332.0 Paralysis agitans 9.9 months ago 332.0 Paralysis agitans 4.1 months ago 332.0 Paralysis agitans now
Predicted vs. actual medication classes for Case 3. Table structure as in case 1.
Top predictions Prob. True labels Prob.
N04B Dopaminergic agents 97.66% C10A Lipid modifying agents, plain 13.90% N03A Antiepileptics 34.01% C09A Ace inhibitors, plain 9.21% N02B Other analgesics and antipyretics 32.81% C01E Other cardiac preparations 5.56% N06A Antidepressants 26.10% C02C Antiadrenergic agents, peripherally acting 0.72% N02A Opioids 20.33% G03B Androgens 0.32%
A14A Anabolic steroids 0.08%
Medication predictions for a patient with only one ICD-9 code, repeated many times over five years. The medications listed under true labels are not indicated for paralysis agitans (Parkinson’s disease), but the patient was surely taking them for reasons not documented in the ICD-9 sequence. The model predicted mostly reasonable medications for a patient with Parkinson’s disease, especially Dopaminergic agents, which is the primary treatment for the disease. Figure representation as in case 1, above.

In light of the detailed author responses and further updates to the manuscript, I am raising my score to an 8 and reiterating my support for this paper. I think it will be among the strongest non-traditional applied deep learning work at ICLR and will receive a great deal of interest and attention from attendees.
-----
This paper describes modern deep learning approach to the problem of predicting the medications taken by a patient during a period of time based solely upon the sequence of ICD-9 codes assigned to the patient during that same time period. This problem is formulated as a multilabel sequence classification (in contrast to language modeling, which is multiclass classification). They propose to use standard LSTM and GRU architectures with embedding layers to handle the sparse categorical inputs, similar to that described in related work by Choi, et al. In experiments using a cohort of ~610K patient records, they find that RNN models outperform strong baselines including an MLP and a random forest, as well as a common sense baseline. The differences in performance between the recurrent models and the MLP appear to be large enough to be significant, given the size of the test set.
Strengths:
- Very important problem. As the authors point out, two the value propositions of EHRs -- which have been widely adopted throughout the US due to a combination of legislation and billions of dollars in incentives from the federal government -- included more accurate records and fewer medication mistakes. These two benefits have largely failed to materialize. This seems like a major opportunity for data mining and machine learning.
- Paper is well-written with lucid introduction and motivation, thorough discussion of related work, clear description of experiments and metrics, and interesting qualitative analysis of results.
- Empirical results are solid with a strong win for RNNs over convincing baselines. This is in contrast to some recent related papers, including Lipton & Kale et al, ICLR 2016, where the gap between the RNN and MLP was relatively small, and Choi et al, MLHC 2016, which omitted many obvious baselines.
- Discussion is thorough and thoughtful. The authors are right about the kidney code embedding results: this is a very promising result.
Weaknesses:
- The authors make several unintuitive decisions related to data preprocessing and experimental design, foremost among them the choice NOT to use full patient sequences but instead only truncated patient sequences that each ends at randomly chosen time point. This does not necessarily invalidate their results, but it is somewhat unnatural and the explanation is difficult to follow, reducing the paper's potential impact. It is also reduces the RNN's potential advantage.
- The chosen metrics seem appropriate, but non-experts may have trouble interpreting the absolute and relative performances (beyond the superficial, e.g., RNN score 0.01 more than NN!). The authors should invest some space in explaining (1) what level of performance -- for each metric -- would be necessary for the model to be useful in a real clinical setting and (2) whether the gaps between the various models are "significant" (even in an informal sense).
- The paper proposes nothing novel in terms of methods, which is a serious weakness for a methods conference like ICLR. I think it is strong enough empirically (and sufficiently interesting in application) to warrant acceptance regardless, but there may be things the authors can do to make it more competitive. For example, one potential hypothesis is that higher capacity models are more prone to overfitting noisy targets. Is there some way to investigate this, perhaps by looking at the kinds of errors each model makes?
I have a final comment: as a piece of clinical work, the paper has a huge weakness: the lack of ground truth labels for missing medications. Models are both trained and tested on data with noisy labels. For training, the authors are right that this shouldn't be a huge problem, provided the label noise is random (even class conditional isn't too big of a problem). For testing, though, this seems like it could skew metrics. Further, the assumption that the label noise is not systemic seems very unlikely given that these data are recorded by human clinicians. The cases shown in Appendix C lend some credence to this assertion: for Case 1, 7/26 actual medications received probabilities < 0.5. My hunch is that clinical reviewers would view the paper with great skepticism. The authors will need to get creative about evaluation -- or invest a lot of time/money in labeling data -- to really prove that this works.
For what it is worth, I hope that this paper is accepted as I think it will be of great interest to the ICLR community. However, I am borderline about whether I'd be willing to fight for its acceptance. If the authors can address the reviewers' critiques -- and in particular, dive into the question of overfitting the imperfect labels and provide some insights -- I might be willing to raise my score and lobby for acceptance.


RECURRENT MIXTURE DENSITY NETWORK FOR SPATIOTEMPORAL VISUAL ATTENTION
In many computer vision tasks, the relevant information to solve the problem at hand is mixed with irrelevant, distracting information. This has motivated researchers to design attentional models that can dynamically focus on parts of images or videos that are salient, e.g., by down-weighting irrelevant pixels. In this work, we propose a spatiotemporal attentional model that learns where to look in a video directly from human fixation data. We model visual attention with a mixture of Gaussians at each frame. This distribution is used to express the probability of saliency for each pixel. Time consistency in videos is modeled hierarchically by: 1) deep 3D convolutional features to represent spatial and short-term time relations at clip level and 2) a long short-term memory network on top that aggregates the clip-level representation of sequential clips and therefore expands the temporal domain from few frames to seconds. The parameters of the proposed model are optimized via maximum likelihood estimation using human fixations as training data, without knowledge of the action in each video. Our experiments on Hollywood2 show state-of-the-art performance on saliency prediction for video. We also show that our attentional model trained on Hollywood2 generalizes well to UCF101 and it can be leveraged to improve action classification accuracy on both datasets.
1 INTRODUCTION
Attentional modeling and saliency prediction in images has been an active research topic in computer vision over the last decade. Interest in attentional models is primarily motivated by their ability to eliminate or down-weight pixels that are not important for the task at hand, as for example shown in prior work using visual attention for image recognition and caption generation (Sermanet et al., 2014; Xu et al., 2015; Mnih et al., 2014). Integrating visual attention in an image analysis model can potentially lead to improved overall accuracy, as the system can focus on the most salient regions in the photo without being disturbed by irrelevant information.
Recently, we have witnessed a shift of trend from image saliency prediction (Borji & Itti, 2013) to the modeling of saliency in videos (Rudoy et al., 2013). Since human fixation patterns are strongly correlated over time (Coull, 2004), it appears critical to model the relations between saliency maps of consecutive frames. In this scenario, attention can be defined as a spatiotemporal volume, where each saliency map (one for each frame) depends on the frames at the previous times. The saliency map can be interpreted as a probability distribution over pixels and the actual fixation patterns can be generated by sampling from the the map.
∗This work was done when Loris Bazzani was at Dartmouth College. †Hugo Larochelle is now at Google Brain.
Going from images to videos is not straightforward, since videos bring up many challenges. First of all, videos have an additional dimension (time), compared to images. This causes a dramatic growth in the number of pixels to be processed and poses a significantly higher computational cost for analysis. At the same time, there are strong redundancies present in such data, which implies that visual attention may be particularly beneficial for the video setting. For example, typically the objects or people in a video do not change significantly in appearance over time. Yet, for analysis tasks such as action recognition (Wang & Schmid, 2013) or video description (Yao et al., 2015), it is imperative to properly model the dynamical properties of these objects in the video. This suggests that, in order to identify spatiotemporal volumes that are salient for video analysis, an attentional model must take into account high-level image semantics as well as the history of past fixations.
In order to cope with these challenges, we propose an efficient spatiotemporal attentional model (see Fig. 1) that leverages deep 3D convolutional features (Tran et al., 2015) as semantic, spatiotemporal representation of short clips in the video. This clip-level representation is then aggregated by a Long Short-Term Memory (LSTM) network (Hochreiter & Schmidhuber, 1997), that expands the temporal range of analysis from few frames to seconds. The LSTM model connects into a Mixture Density Network (MDN) (Bishop, 1994) that at each frame outputs the parameters of a Gaussian mixture model expressing the saliency map. We refer to this model as Recurrent Mixture Density Network (RMDN). RMDN is trained via maximum likelihood estimation using human fixations as training data, without knowledge of the actions in the videos.
The potential applications of automatic saliency map prediction from videos are many. They include attention-based video compression (Gitman et al., 2014), visual attention for robots (Yu et al., 2010), crowd analysis for video surveillance (Jiang et al., 2014), salient object detection (Li & Yu, 2015; Karthikeyan et al., 2015) and activity recognition (Vig et al., 2012; Sapienza et al., 2014). In this work we focus on a study of how visual attention may improve action recognition by leveraging the saliency map generated by RMDN for video classification. The idea is akin to soft attention and consists in re-weighting the pixel values of the input video by the estimated saliency map. Despite its simplicity, we show that the combination of features extracted from this modified version of the video and those computed from the original input lead to a significant improvement in action recognition, compared to a model that does not use attention.
The primary contribution of this work is a spatiotemporal saliency estimation network optimized to reproduce human fixations. The proposed approach offers several advantages: 1) the model can be trained without having to engineer spatiotemporal features; 2) RMDN is directly trained on examples of human fixations and thus learns to mimic human visual attention; 3) prediction of the saliency map is very fast (it takes 0.08s per 16-frame clip on a GPU); 4) the method outperforms the state-of-the-art (Mathe & Sminchisescu, 2015) in saliency accuracy; 5) our predicted saliency maps lead to to improvements in action classification accuracy.
2 RELATED WORK
Broadly speaking, the literature on attentional models can be split into two categories: task-agnostic approaches which model the bottom-up, free-viewing properties of attention, and task-specific methods which model its top-down, task-driven properties. Researchers have devoted many years to create datasets, collecting human fixations and proposing solutions for biologically-plausible saliency estimators, built using low-level cues such as edge detectors and color filters (e.g. see Borji et al. (2013); Judd et al. (2009); Harel et al. (2006) for recent examples). We refer to Borji & Itti (2013) and Bruce et al. (2016) for an interesting analysis and comparison of existing methods. Most of the techniques in the literature are focused on extracting features in a bottom-up and/or top-down manner and use them to estimate the saliency map. In this context, motion features are introduced when extending saliency methods from images to videos (Guo et al., 2008; Zhao et al., 2015; Zhai & Shah, 2006). However, there is no explicit modeling of the temporal dimension that can capture long-term relations. In fact, motion features (e.g., optical flow) describe short-term associations at the temporal scale of only a few consecutive frames.
Prior approaches can also be categorized into soft-attentional versus hard-attentional models. Softattentional models use the predicted saliency maps to down-weight pixels that are not relevant or salient, e.g., Song et al. (2016). Specifically deep networks have been used in this context to assign a weight to each pixel in order to extract “glimpses” from images (Xu et al., 2015; Gregor et al., 2015)
or videos (Yao et al., 2015) in the form of weighted pixel averages. One strength of such approaches is that they can backpropagate through the attentional component and tune it in the context of its use in a deep network. Other work has been geared towards learning hard-attentional models, which explicitly ignore and discard parts of the input (Larochelle & Hinton, 2010; Bazzani et al., 2011; Denil et al., 2012; Mnih et al., 2014; Xu et al., 2015; Sermanet et al., 2014; Ba et al., 2015; Yoo et al., 2015; Zheng et al., 2015), thus providing significant computational savings. Unfortunately, such models are often hard to train because they rely on reinforcement learning techniques to generate the image/video locations during training.
All of the aforementioned prior work attempts to learn attentional models indirectly rather than from explicit information of where humans look. Recent work (Mauthner et al., 2015; Hossein Khatoonabadi et al., 2015; Mathe & Sminchisescu, 2015; Stefan Mathe, 2013; Kümmerer et al., 2015; Rudoy et al., 2013) has shown that it may be possible to accurately reproduce gazing patterns of human subjects attending to images and videos. However, these prior approaches rely on hand-crafted features to estimate the saliency maps. Attempts at removing hand-engineering of features are represented by Jetley et al. (2016); Huang et al. (2015); Kümmerer et al. (2015) where networks pre-trained for object recognition were subsequently finetuned using saliency-based loss functions for images. Pan & i Nieto (2015) followed the same principle but without using any pre-trained network for initialization. Liu et al. (2015) proposed a multi-scale architecture for saliency prediction, and Li & Yu (2015) added a refinement step in order to enforce spatial coherence of the output. Simonyan et al. (2014) and Mahendran & Vedaldi (2016) proposed to reverse deep networks using deconvolutions for visualization and to estimate image saliency. However, these methods estimate saliency from still images and do not consider the temporal aspect of video. Chaabouni et al. (2016a;b) trained a ConvNet for saliency prediction on optical flow features and individual frames. However the model uses only the very short-term temporal relations of two consecutive frames.
In this paper, we explore the following question: can deep networks be trained to reliably predict spatiotemporal attentional patterns, specifically in such a way that these predictions can be leveraged successfully by a recognition system? To our knowledge, our work distinguishes itself from the aforementioned literature by being the first application of deep networks to the prediction of spatiotemporal human saliency in videos.
3 PROPOSED MODEL
We start with a high-level description of our attentional model. We then formalize it in Sec. 3.1, and describe its training in Sec. 3.2. Sec. 3.3 reports how prediction is efficiently carried out at test time. Sec. 3.4 describes how to leverage the predicted saliency map to improve action recognition.
The proposed RMDN model for saliency estimation is depicted in Fig. 1. At time t, the input of the model is a sequence of the lastK = 16 frames, i.e., from time t−K+1 to current time t. We refer to this sequence as the input “clip.” The first part of the model (Fig. 1, blue layers above the input clip) consists of a 3D convolutional network that provides a feature representation of the clip. Our choice of a clip-based representation rather than a single-frame descriptor is motivated by the fact that these features allow us to explicitly capture short-term information that is then aggregated for long-term spatiotemporal visual attention by RMDN. Furthermore, there is recently growing evidence (Tran et al., 2015; Srivastava et al., 2015; Yue-Hei Ng et al., 2015) that by modeling the temporal information it is possible to obtain improved performances in high-level video analysis tasks, such as action recognition. For the computation of spatiotemporal features from the input clip, we use the “C3D” architecture proposed by Tran et al. (2015), which has been shown to provide competitive results on video recognition tasks across different datasets. The C3D architecture is defined as: C64-P-C128-P-C256-C256-P-C512-C512-P-C512-C512-P-FC4096-FC4096-softmax, where C is a 3D convolutional layer, P is the pooling layer, FC is a fully-connected layer, and the number specifies the number of kernels of the layer (e.g. C64 has 64 kernels). For the details about the size and stride of the convolutional and pooling kernels, we refer to (Tran et al., 2015).
The convolutional network has access to a limited window of the video since it uses a fixed-size clip of 16 frames as input. In order to empower the visual attention model with the ability to take into account longer temporal extents, we need a mechanism that performs temporal aggregation of past clip-level signals. To this end, we propose to connect the internal representation of the C3D model to a recurrent neural network, as shown in Fig. 1 (green module). The aim of the temporal
connections of the recurrent neural network is to propagate the clip-level features through time via memory units that can capture long-term dependencies. Our model uses LSTMs (Hochreiter & Schmidhuber, 1997) as memory blocks.
The saliency map at each time t is expressed in terms of a Gaussian Mixture Model (GMM) with C components. We denote its parameters with {(µc, πc,Σc)}Cc=1, where µc, πc and Σc are the mean, the mixture coefficient and the covariance of the c-th Gaussian component, respectively. The LSTM directly outputs these parameters (see details below). The resulting network is known as a Mixture Density Network (MDN) (Bishop, 1994; Graves, 2013).
Since the model is recurrent, there is a direct connection between the inner representation of the LSTM at time t and the one at time t + 1. This favors temporal consistency between the saliency maps at adjacent times.
3.1 FORMALIZATION OF THE MODEL
Let D = {(vi,ai)}Ni=1 be a dataset of videos and human fixations pairs. vi = (cit) Ti−1 t=0 is a video consisting of Ti temporally overlapping clips cit (i.e., sampled with stride 1) and a i = (ait) Ti−1 t=0 is the sequence of ground-truth fixations for the i-th video aligned with the clips. Since we use C3D to represent each clip, cit has a fixed length of K = 16 frames and t = 0 means that the first K frames are used to build the 0-th clip. The fixations ait = {ait,j}Aj=1 are a set of (x, y) image positions that are normalized to [0, 1] in order to deal with videos of different resolutions. The number of fixations vary from frame to frame in general, but in our experiments we control it via subsampling in order to obtain for each frame a set of fixed size A.
Let xt = C3D(ct) be the internal representation of C3D for an input clip ct. In our model we use the last convolutional layer, before the fully-connected layers. We choose a convolutional layer instead of a fully-connected layer because the latter discards spatial information, which is crucial to estimate a spatially-variant saliency map over the image.
The LSTM network (Hochreiter & Schmidhuber, 1997) is defined as follows:
ft = σ(Wf · [ht−1,xt] + bf ), it = σ(Wi · [ht−1,xt] + bi) (1) ot = σ(Wo · [ht−1,xt] + bo), C̃t = tanh (WC · [ht−1,xt] + bC) (2) Ct = ft ∗ Ct−1 + it ∗ C̃t, ht = ot ∗ tanh (Ct) (3)
where ft, it, ot, Ct and ht are the forget gate, the input gate, the output gate, the memory cell, and the hidden representation, respectively. The learning parameters that need to be estimated during the training phase are Wz and bz ,where z ∈ {f, i, o, C}. The MDN (Graves, 2013; Bishop, 1994) takes its inputs from the hidden representation of the LSTM network. Since the output space is 2D (the space of image locations), we can reparametrize the model as {(µct , πct , σct , ρct)}Cc=1, where µct , πct , σct and ρct are the 2D mean position, the weight, the 2D variance and the correlation of the c-th Gaussian component, respectively. The MDN is therefore defined as follows:
yt = {(µ̃ct , π̃ct , σ̃ct , ρ̃ct)}Cc=1 = Wy · ht + by (4) where Wy and by are the parameters of the linear layer and ht is the hidden representation of the LSTM network. The parameters of the GMM in Eq. 4 are normalized as follows in order to obtain a valid probability distribution:
µct = µ̃ c t , π c t = exp(π̃ct )∑C i=1 exp(π̃ i t) , σct = exp(σ̃ c t ), ρ c t = tanh (ρ̃ c t). (5)
The composition of the LSTM and the MDN results in the RMDN.
3.2 TRAINING
The proposed model can be trained by optimizing the log-likelihood of the training ground truth fixations, ai, under the GMM. The loss function for the i-th video, vi, is defined as the negative log-likelihood of the fixations under the GMM, as follows:
L(vi,ai) = Ti−1∑ t=0 A∑ j=1 − log
( C∑
c=1
πctN (ait,j ;µct , σct , ρct)
) (6)
whereN is the Gaussian distribution. Note that the parameters of the Gaussian components depend on the input video vi, but we do not make this explicit in the equation in order to keep notation simple.
The log-likelihood of the RMDN is optimized using backpropagation through time, since it is a composition of continuous functions (e.g. linear transformations and element-wise non-linearities) and the LSTM, for which we can compute the gradients. In particular, we refer to the paper of Graves (2013) for the derivation of the gradients for the MDN using the loss function of Eq. 6. In practice, due to the limited training data, we freeze the layers of the C3D network to the values pretrained by Tran et al. (2015) for action recognition. This implies that the low-level representation xt is fixed. We jointly train the LSTM and MDN from randomly initialized parameters.
3.3 PREDICTION
The inference stage is straightforward by following the equations of Sec. 3.1. At a given time t, the clip from time t − K + 1 to t is fed into the C3D network to produce the representation xt. This vector is passed to the LSTM (Eqs. 1, 2, and 3) whose hidden representation is passed to the MDN, which outputs the GMM parameters (Eqs. 4 and 5). In order to generate the final saliency map, we compute the probability of each pixel position under the GMM model. We normalize the probability map to sum up to 1 over the image pixels in order to produce a normalized saliency map.
3.4 SALIENCY FOR CLASSIFICATION
For the task of video classification, we generate a modified version of the video by using a softattentional mechanism: the idea is to weight each pixel value by the estimated saliency at that
position. This operation effectively down-weights regions that are deemed not salient. The intuition is that then the classifier will be able to focus on the parts of the frame which are most relevant without being distracted by the non-salient regions (see Fig. 2 in Appendix A).
At each time t, we extract two representations: the “context” branch is given by the C3D representation of the original clip, while the “soft attentional” branch is given by the C3D representation of the input clip weighted by the saliency map. The rationale is that the context branch considers the global evolution of the activity in the video while the soft attentional branch is focused on the most-salient local evolution of the activity. The two representations are then concatenated at the clip level and max-pooled over the video to obtain the final video-level descriptor. This video-level representation is then used as input to train the video classifier, which is a linear SVM in all our experiments.
4 EXPERIMENTS
In this section, we evaluate the proposed method for both saliency prediction and action recognition on two challenging datasets: Hollywood2 (Marszałek et al., 2009) and UCF101 (Soomro et al., 2012). Section 4.1 reports a quantitative analysis for the task of saliency prediction. Section 4.2 shows the results for the action recognition task in two scenarios: 1) using the same dataset that was used to train the saliency predictor and 2) applying the pretrained attentional model to a neverseen dataset and a different set of actions. We reported the implementation details in Appendix B. We invite the reader to watch the qualitative results of the proposed method in the form of a video available at https://youtu.be/aXOwc17nx_s.
4.1 SALIENCY PREDICTION
The proposed model is trained using human fixation data. Few datasets provide both human fixations and class labels, which we need for the action recognition experiment discussed in the next section. Therefore, we used the Hollywood2 dataset, which was augmented with eye tracking data by Mathe & Sminchisescu (2015). We follow the same evaluation protocol (i.e., same training/test splits) of Mathe & Sminchisescu (2015) and their validation procedure to compute the final results in order to compare with their work. Mathe & Sminchisescu (2015) generate the ground truth saliency from a binary fixation map where the only non-zero values are at fixations points. The final saliency map is produced by convolving the binary map with an isotropic Gaussian filter with standard deviation σ and then adding to it a uniform distribution with probability parameter p. As in Mathe & Sminchisescu (2015), the values of these two parameters are chosen from σ ∈ {1.5, 3.0} and p ∈ {0.25, 0.5} via hold-out validation. We use a validation set consisting of 20% of the training set. We use the remaining 80% of the training data to learn our models, and use the hold-out validation set to choose the hyperpameters of our model.
We evaluate all models on the test set, using popular metrics proposed in the literature of saliency map prediction for still images (Judd et al., 2012; Borji et al., 2013), such as Area Under the ROC Curve (AUC), Normalized Scanpath Saliency (NSS), linear Correlation Coefficient (CC) and the Similarity score (Sim). We refer to the papers of Judd et al. (2012) and Borji et al. (2013) for their detailed description.
Table 1 shows results achieved with different variants of our model and a simple baseline method, which we refer to as Trained Central Bias (TCB). The TCB model is a single GMM trained using the fixations of all the videos in the training sets. TCB predicts the same saliency map for each testing frame, thus it discards completely the temporal information and the image input. This experiment shows that all versions of our RMDN consistently outperform TCB under all metrics, even when using a smaller number of fixations per frame during training.
The different variants of RMDN in Table 1 explore the following design choices in our model: 1) the impact of using LSTM hidden units as opposed to regular RNN units (second and third row) and 2) the number of fixations per frame used for training (third and fourth row). These experiments show that LSTM (third row) is better than an RNN (second row) in terms of AUC and NSS, but in order to have better CC and Sim we need to use more fixations per frame (fourth row). This is intuitive: since the LSTM has many more parameters than the RNN, it needs more training data to be properly optimized.
The last row in Table 1 shows the results obtained by retraining our model using the full training set of Mathe & Sminchisescu (2015) instead of just the 80% subset. For this case (RMDN full) we used the hyper-parameter values selected via hold out-validation for the experiment in the fourth row. This gives the best result for saliency prediction reported in our work and it is the model we used for all the subsequent experiments described below.
All of the experiments reported in Table 1 were obtained using C = 20 components in the GMM. We have also studied how the accuracy varies by reducing the value of C. For example, using the RMDN variant of row 5 in Table 1 but with C = 10 components (instead of 20), the performance does not change dramatically, yielding AUC= 0.8966 and NSS= 2.4392. On the other hand, the AUC and the NSS decrease considerably, by 1.3% and 0.3 points respectively, when using only C = 2 components (AUC= 0.8836 and NSS= 2.1385). Based on this analysis, in all our subsequent experiments we used C = 20 as we noticed that our approach implements automatically a sort of Occam’s razor, setting the weights πc of many components close to zero when necessary.
We have also carried out a few side experiments and discovered that using the fully-connected features of C3D instead of the convolutional representation gives results that are at least 1.5% lower in terms of AUC. Moreover, we tried to finetune the C3D network for action categorization on Hollywood2. However we did not obtain any significant improvement, confirming the findings of Tran et al. (2015): the C3D representation is already general enough to perform effectively on different action recognition tasks and fine-tuning the model on smaller-scale datasets (such as Hollywood2) does not seem beneficial. We also experimented with deep LSTMs, but we obtained an insignificant improvement of performance. For this reason and also because deep LSTMs have more parameters and are more computationally expensive to train, we chose to use a shallow one-layer LSTM. Finally, we run the ablation study where the recurrent link between time t − 1 and t of the RMDN is removed: the results in terms of AUC are 1.2% and 2.4% lower with respect to the RMDN which uses RNN (second row of Table 1) and LSTM (third row of Table 1), respectively.
We also compared our approach to the state-of-the-art in saliency prediction from video. Table 2 includes the results of the best methods taken from the extensive analysis done in Mathe & Sminchisescu (2015). The table reports also some useful baselines, such as the central bias (CB) and the human accuracy for the task. Note that: 1) CB differs from TCB, since it does not use any training fixations; and 2) the human accuracy is computed in (Mathe & Sminchisescu, 2015) by deriving a saliency map from half of our human subjects and is evaluated with respect to fixations of the remaining ones. Furthermore, Table 2 contrasts the use of static features, motion features and their combination. The last row reports the results obtained with our RMDN model. It is interesting to see that the results obtained with a single type of features (static or motion) have an AUC lower than 0.75, which is even lower than the one obtained by the central bias (0.84). Moreover, the combination reaches the best results when the central bias is combined with engineered features (SF+MF+CB). On the other hand, our method outperforms all the methods evaluated in Mathe & Sminchisescu (2015) by a large margin and our results are very close to human performance (the difference is only 3.2%). In addition to being the best method in Table 2, our method has several advantages: 1) it does not require any hand-engineering of spatiotemporal features, 2) it performs joint training of the LSTM and the saliency predictor, 3) it is very efficient. Specifically, although we cannot estimate the runtime for prior approaches, we believe that our method is much faster than most of the methods reported in Table 2 as these depend on features that are computationally expensive to extract. Our proposed method takes only 0.08s per clip for inference on GPU: 0.07s to compute C3D features and 0.01s to evaluate the RMDN.
4.2 ACTION RECOGNITION
In order to show how saliency can be used for action recognition we carried out a set of experiments covering two scenarios: 1) using the same dataset where the saliency predictor was trained (Hollywood2) and 2) using a never-seen dataset with a different set of actions (UCF101).
The results on Hollywood2 are reported in terms of mean Average Precision (mAP) as done by Mathe & Sminchisescu (2015). Table 3 shows an analysis of 1) the impact of using different feature representations as well as 2) the effect of the saliency map. As in Tran et al. (2015), we experimented with different features, namely CONV5 and FC6, which correspond to the fifth convolutional layer and the first fully-connected representation of C3D, respectively. We also tested two ways to use the saliency maps, called in the second column: “feature” and “clip”. In the feature mode (first row, experiments (2-5)), the convolutional representation is multiplied by the saliency map, after resizing it accordingly. In other words, the saliency weights directly the feature representation, similarly to the work of Sharma et al. (2016). In the clip mode (second and third row, experiments (2-5)), we adopted the model presented in Sec. 3.4, where the saliency maps are used to weight the input video pixels.
The third column of Table 3 (experiment (1)) reports the results using only the original video as input to C3D (referred to as context in Fig. 2). Experiment (2) uses the ground truth saliency maps as soft attention to weight the input of C3D, while in experiment (3) this vector is concatenated with the context features. The last two columns (experiment (4) and (5)) represent the same setup, but in this case we use the saliency maps predicted by our model instead of the ground truth.
Table 3 shows that the results of CONV5 and FC6 are very close when considering the original video (experiment (1)). The table also shows that the feature mode has lower performance compared to the clip mode (experiment (2)). Moreover, the concatenation (experiment (3)) is effective only when visual attention is used to weight pixels rather than features. Based on the poor performance of the features mode, we decided to experiment only with the clip mode in our study with predicted saliency (experiment (4) and (5)). Also, we decided to use FC6 features for the rest of the paper because the representation is more compact and therefore allows to train the classifiers more quickly. We can notice that even in the case of predicted saliency, the concatenation of FC6 context features and those obtained by weighting the input video with soft-attention (experiment (5)) produces a significant improvement over the original CONV5/FC6 features without attention. Furthermore, a pleasant surprise is represented by the the small difference in results between using the predicted saliency (experiment (5)) versus the ground truth maps (experiment (3)): only 0.27% for FC6 (last row).
The SVM model complexity for experiment (3) in Table 3 is twice as large as the complexity for (1) and (2) since the feature dimensionality is doubled by construction. The same applies when comparing (5) against (1) and (4). In order to have a more fair comparison, we added PCA dimensionality reduction to experiment (5) in order to match the same feature dimensionality as (1) (and (4)). Although the validation accuracies are very similar, the testing mAP drops from 54.85% of experiment (5) to 51.82% of the PCA experiment. This is not surprising, since the extra dimensions
provided by the use of the saliency map are not redundant with respect to the context representation (1). Therefore, concatenation seems to be an effective way to make use of the saliency map.
Table 4 compares our action categorization results with those presented in Mathe & Sminchisescu (2015). As we did before, we separate experiments that use the ground truth maps and those that use predicted saliency. The results of Table 4 show that the performance our method (second and fifth column) is around 2% lower than Mathe & Sminchisescu (2015). However this is most likely explained by the differences in the type of features and classifier, and not by the differences in saliency map prediction methods. Indeed, we already established in Table 2 that our proposed saliency map predictor is more accurate than the one proposed in Mathe & Sminchisescu (2015). On the other hand, Mathe & Sminchisescu (2015) use a combination of many different features and a kernel chi-square SVM, while our method uses C3D features with a simple linear SVM classifier. Adding more non-linearities, especially for the concatenation experiment, would probably help. But we consider the experimentation with different types of action recognition features and classifiers out of the scope of this paper.
Finally, we perform an experiment to assess the generalization abilities of the learned saliency model to a different dataset, with classes and videos that have not been seen during its training. To this end, we used the attentional model trained on the Hollywood2 dataset to extract saliency maps on the UCF101 dataset. As saliency ground truth is not available for UCF101, we evaluate performance in terms of action recognition accuracy using the evaluation protocol and splits by Soomro et al. (2012). Table 5 summarizes the results. The proposed method (C3D + RMDN, eight row) corresponds to the concatenation of the original C3D descriptor and the C3D descriptor with the input weighted by the saliency map, as was done in the Hollywood2 experiments. We compare our method with the results obtained using the C3D descriptor computed from the context only (seventh row) and other state-ofthe-art methods (first row through sixth row). A linear SVM trained on C3D features computed from the context already outperforms most of the other methods (first row to forth row). But training the linear SVM on a concatenation of context C3D features and those obtained by reweighting the video
input with the RMDN saliency maps (seventh row) leads to a further improvement of 1.1%. This is an impressive result since the RMDN was trained on the separate and small Hollywood2 dataset.
Since we noticed that the saliency maps of RMDN for UCF101 tend to be highly peaked around a single location in each frame, we added the trained central bias (already analyzed in Table 1). This has the effect of diffusing the saliency map with the central bias, thus enlarging the area of attention used by the recognition system. The result of this experiment, which is reported in the last row of Table 5, further improves the accuracy by 1.3%.
5 CONCLUSIONS
In this paper, we proposed a recurrent mixture density network for spatiotemporal visual attention. We showed that our model outperforms state-of-the-art methods for saliency prediction in videos. We have also shown that the saliency maps generated by our model can be leveraged to improve action categorization using a very simple procedure. This suggests that saliency can enrich the original video representation. The runtime overhead to estimate the saliency map is very small: only 0.01s added to the feature extraction time of 0.07s.
As future work, we plan to close the gap between RMDN and action recognition with a joint network. The idea is to have as output of the model both the saliency map at each time and the class of the action for the entire video. This can be combined with the idea of using the saliency map estimated at the previous time to weight the input for the current time. Putting together these two ideas in a single network would result in a joint model for saliency prediction and action recognition.
6 ACKNOWLEDGMENTS
We thank Du Tran for helpful discussion about the code of the C3D network and its usage. We are grateful to Stefan Mathe for explaining the format of the eyetracking data and the protocol of the Hollywood2 experiment. This work was funded in part by NSF award CNS-1205521. We gratefully acknowledge NVIDIA for the donation of GPUs used for portions of this work.
A SALIENCY FOR CLASSIFICATION
The proposed model for recognition is presented in Fig. 2. At each time t, we extract two representations: the context branch is given by the C3D representation of the original clip, while the soft attentional branch is given by the C3D representation of the input clip weighted by the saliency map. The two representations are then concatenated at the clip level and max-pooled over the video to obtain the final video-level descriptor. This video-level representation is then used as input to train the video classifier which is a linear SVM in our experiments.
In our experiments, we also evaluated the option of weighting the convolutional feature map xt instead of the input, as for example done by Sharma et al. (2016). However, we will see that soft-masking the input gives higher accuracy, probably because applying C3D’s non-linear transformation after the soft-weighting produces a representation that is less redundant with the original (non-masked) C3D representation.
B IMPLEMENTATION DETAILS
We used the pretrained C3D network (Tran et al., 2015) as feature representation which is the input of the LSTM network. The convolutional layer before the fully-connected layers is used for saliency prediction, while the last fully-connected layer before the softmax is used for classification, since Tran et al. (2015) showed to obtain the best performance.
The training of the RMDN is performed using RMSprop with adaptive learning rate and gradient clipping. We start from a learning rate of 0.0003 and after 8 epochs it is reduced at each epoch with a decay factor of 0.95. The gradient is clipped with a threshold of 20. Dropout with a ratio of 0.5 is applied only on the hidden layer of the LSTM network before the MDN. We trained for 40 epochs, but training is stopped if there is no significative improvement of the loss. During training, temporal data augmentation is performed by clipping the videos to shorter videos of length 65 frames (which corresponds to 50 C3D descriptors since it needs a buffer of 16 frames for the first descriptor). The number of components of the GMM C is fixed to 20 for all the experiments. All the experiments were carried out using an NVIDIA Tesla K40 card.
After extracting the saliency maps and the feature representations on GPU, our recognition experiments were performed on CPU using a linear SVM. In order to compute the video-level representation, we performed max pooling of the clip-level representations of the video. For all the experiments, we used 20% of the training data as validation set to find the regularization parameter of the SVM. We searched the parameter space on a grid between 10−9 to 103 with a step of 10 1 2 . Finally, we retrain the SVM on all the training set (including the validation set) using the best cross-validated parameter.

This work proposes to a spatiotemporal saliency network that is able to mimic human fixation patterns,
thus helping to prune irrelevant information from the video and improve action recognition.
The work is interesting and has shown state-of-the-art results on predicting human attention on action videos.
It has also shown promise for helping action clip classification.
The paper would benefit from a discussion on the role of context in attention.
For instance, if context is important, and people give attention to context, why is it not incorporated automatically in your model?
One weak point is the action recognition section, where the comparison between the two (1)(2) and (3) seems unfair.
The attention weighted feature maps in fact reduce the classification performance, and only improve performance when doubling the feature and associated model complexity by concatenating the weighted maps with the original features.
Is there a way to combine the context and attention without concatenation?
The rational for concatenating the features extracted from the original clip,
and the features extracted from the saliency weighted clip seems to contradict the initial hypothesis that `eliminating or down-weighting pixels that are not important' will improve performance.
The authors should also mention the current state-of-the-art results in Table 4, for comparison.
# Other comments:
# Abstract
- Typo: `mixed with irrelevant ...'
``Time consistency in videos ... expands the temporal domain from few frames to seconds'' - These two points are not clear, probably need a re-write.
# Contributions
- 1) `The model can be trained without having to engineer spatiotemporal features' - you would need to collect training data from humans though..
# Section 3.1
The number of fixation points is controlled to be fixed for each frame - how is this done?
In practice we freeze the layers of the C3D network to values pretrained by Tran etal.
What happens when you allow gradients to flow back to the C3D layers?
Is it not better to allow the features to be best tuned for the final task?
The precise way in which the features are concatenated needs to be clarified in section 3.4.
Minor typo:
`we added them trained central bias'