Sepsis is a life-threatening complication from infection and a leading cause of mortality in hospitals. While early detection of sepsis improves patient outcomes, there is little consensus on exact treatment guidelines, and treating septic patients remains an open problem. In this work we present a new deep reinforcement learning method that we use to learn optimal personalized treatment policies for septic patients. We model patient continuous-valued physiological time series using multi-output Gaussian processes, a probabilistic model that easily handles missing values and irregularly spaced observation times while maintaining estimates of uncertainty. The Gaussian process is directly tied to a deep recurrent Q-network that learns clinically interpretable treatment policies, and both models are learned together end-to-end. We evaluate our approach on a heterogeneous dataset of septic spanning 15 months from our university health system, and find that our learned policy could reduce patient mortality by as much as 8.2% from an overall baseline mortality rate of 13.3%. Our algorithm could be used to make treatment recommendations to physicians as part of a decision support tool, and the framework readily applies to other reinforcement learning problems that rely on sparsely sampled and frequently missing multivariate time series data.
This paper presents an important application of modern deep reinforcement learning (RL) methods to learning optimal treatments for sepsis from past patient encounters. From a methods standpoint, it offers nothing new but does synthesize best practice deep RL methods with a differentiable multi-task Gaussian Process (GP) input layer. This means that the proposed architecture can directly handle irregular sampling and missing values without a separate resampling step and can be trained end-to-end to optimize reward -- patient survival -- without a separate ad hoc preprocessing step. The experiments are thorough and the results promising. Overall, strong application work, which I appreciate, but with several flaws that I'd like the authors to address, if possible, during the review period. I'm perfectly willing to raise my score at least one point if my major concerns are addressed.
QUALITY
Although the core idea is derivative, the work is executed pretty well. Pros (+) and cons (-) are listed below:
+ discussion of the sepsis application is very strong. I especially appreciated the qualitative analysis of the individual case shown in Figure 4. While only a single anecdote, it provides insight into how the model might yield clinical insights at the bedside.
+ thorough comparison of competing baselines and clear variants -- though it would be cool to apply offline policy evaluation (OPE) to some of the standard clinical approaches, e.g., EGDT, discussed in the introduction.
- "uncertainty" is one of the supposed benefits of the MTGP layer, but it was not at all clear how it was used in practice, other than -- perhaps -- as a regularizer during training, similar to data augmentation.
- uses offline policy evaluation "off-the-shelf" and does not address or speculate the potential pitfalls or dangers of doing so. See "Note on Offline Policy Evaluation" below.
- although I like the anecdote, it tells us very little about the overall policy. The authors might consider some coarse statistical analyses, similar to Figure 3 in Raghu, et al. (though I'm sure you can come up with more and better analyses!). 
- there are some interesting patterns in Table 1 that the authors do not discuss, such as the fact that adding the MGP layer appears to reduce expected mortality more (on average) than adding recurrences. Why might this be (my guess is data augmentation)?
CLARITY
Paper is well-written, for the most part. I have some nitpicks about the writing, but in general, it's not a burden to read.
+ core ideas and challenges of the application are communicated clearly
- the authors did not detail how they chose their hyperparameters (number of layers, size of layers, whether to use dropout, etc.). This is critical for fully assessing the import of the empirical results.
- the text in the figures are virtually impossible to read (too small)
- the image quality in the figures is pretty bad (and some appear to be weirdly stretched or distorted)
- I prefer the X-axis labels that Raghu uses in their Figure 4 (with clinically interpretable increments) over the generic +1, +2, etc., labels used in Figure 3 here
Some nitpicks on the writing
* too much passive voice. Example: third paragraph in introduction ("Despite the promising results of EGDT, concerns arose."). Avoid passive voice whenever possible.
* page 3, sec. 2.2 doesn't flow well. You bounce back and forth between discussion of the Markov assumption and full vs. partial observability. Try to focus on one concept at a time (and the solution offered by a proposed approach). Note that RNNs do NOT relax the Markov assumption -- they simply do an end run around it by using distributed latent representations.
ORIGINALITY
This work scores relatively low in originality. It really just combines ideas from two MLHC 2017 papers [1][2]. One could read those two papers and immediately conclude this paper's findings (the GP helps; RL helps; GP + RL is the best). This paper adds few (if any) new insights.
One way to address this would be to discuss in greater detail some potential explanations for why their results are stronger than those in Raghu and why the MTGP models outperform their simpler counterparts. Perhaps they could run some experiments to measure performance as a function of the number of MC samples (if perhaps grows with the number of samples, then it suggests that maybe it's largely a data augmentation effect).
SIGNIFICANCE
This paper's primary significance is that it provides further evidence that RL could be applied successfully to clinical data and problems, in particular sepsis treatment. However, this gets undersold (unsurprising, given the ML community's disdain for replication studies). It is also noteworthy that the MTGP gives such a large boost in performance for a relatively modest data set -- this property is worth exploring further, since clinical data are often small. However, again, this gets undersold.
One recommendation I would make is that the authors directly compare the results in this paper with those in Raghu and to point out, in particular, the confirmatory results. Interestingly, the shapes of the action vs. mortality rate plots (Figure 4 in Raghu, Figure 3 here) are quite similar -- that's not precisely replication, but it's comforting.
NOTE ON OFFLINE POLICY EVALUATION
This work has the same flaw that Raghu, et al., has -- neither justifies the use of offline policy evaluation. Both simply apply Jiang, et al.'s doubly robust approach [3] "off the shelf" without commenting on its accuracy in practice or discussing potential pitfalls (neither even considers [4] which seems to be superior in practice, especially with limited data). As far as I can tell (I'm not an RL expert), the DR approach carries stronger consistency guarantees and reduced variance but is still only as good the data it is trained on, and clinical data is known to have significant bias, particularly with respect to treatment, where clinicians are often following formulaic guidelines. Can we trust the mortality estimates in Table 1? Why or why not? Why shouldn't I think that RL is basically guaranteed to outperform non-RL approaches under an evaluation that is itself an RL model learned from the same training data!
While I'm willing to accept that this is the best we can do in this setting (we can't just try the learned policy on new patients!), I think this paper (and similar works, like Raghu, et al.) *must* provide a sober and critical discussion of its results, rather than simply applaud itself for getting the best score among competing approaches.
REFERENCES
[1] Raghu, et al. "Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approach." MLHC 2017.
[2] Futoma, et al. "An Improved Multi-Output Gaussian Process RNN with Real-Time Validation for Early Sepsis Detection." MLHC 2017.
[3] Jiang, et al. "Doubly robust off-policy value evaluation for reinforcement learning." ICML 2016.
[4] Thompson and Brunskill. "Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning." ICML 2016.