Individuals with type 1 diabetes (T1D) lack the ability to produce the insulin their bodies need. As a result, they must continually make decisions about how much insulin to self-administer in order to adequately control their blood glucose levels. Longitudinal data streams captured from wearables, like continuous glucose monitors, can help these individuals manage their health, but currently the majority of the decision burden remains on the user. To relieve this burden, researchers are working on closed-loop solutions that combine a continuous glucose monitor and an insulin pump with a control algorithm in an 'artificial pancreas.' Such systems aim to estimate and deliver the appropriate amount of insulin. Here, we develop reinforcement learning (RL) techniques for automated blood glucose control. Through a series of experiments, we compare the performance of different deep RL approaches to non-RL approaches. We highlight the flexibility of RL approaches, demonstrating how they can adapt to new individuals with little additional data. On over 21k hours of simulated data across 30 patients, RL approaches outperform baseline control algorithms (increasing time spent in normal glucose range from 71% to 75%) without requiring meal announcements. Moreover, these approaches are adept at leveraging latent behavioral patterns (increasing time in range from 58% to 70%). This work demonstrates the potential of deep RL for controlling complex physiological systems with minimal expert knowledge.
The paper describes an RL based approach to administer insulin for blood glucose control among  type-1 diabetic patients.  The paper formulates this blood glucose control problem as a closed-loop reinforcement learning problem and demonstrates its effectiveness on data generated from an FDA-approved simulator of glucoregulatory system. Compared to existing approaches, the proposed method can operate without meal announcement by potentially making use of latent meal intake patterns. The authors also demonstrate how a learned policy for one particular subject can be used as initialization to train/fine-tuned the policy of another subject so as to combat the issue of high sample complexity.
Overall, the reviewer finds that the authors provide a reasonable approach to model the blood glucose management problem for type-1 diabetes. Each component of the paper is well-explained and the paper is easy to follow. The algorithmic design adopted in this paper, such as the representation of the problem, the choice of the reward function, and the choice of RL controller, are reasonably justified. While the experiments conducted in this paper are not based on real-world data, the reviewer finds the use of data from an FDA-approved simulator of glucoregulatory system sufficiently convincing for this type of problem. The limitation of the proposed method is also well discussed.
The reviewer has the following concerns:
1. The reviewer views the major contribution of this paper as formulating and solving the glucose management problem as an RL problem.  From a machine learning perspective, the reviewer finds the contribution made in this paper to advance ML methodology very limited as the components used in the algorithmic design of the proposed method are already available in the existing literature. Therefore, the reviewer finds that the paper could be of limited interest to the audience in ICLR while it might be more suitable to the audience of diabetes management.
2. Compared to competing methods reported in this paper, a major characteristic of the proposed method is that it can operate without meal announcements. Methods with meal announcements seem to operate reasonably well compared to the proposed method. Therefore, the authors can consider further justifying why meal announcement is an important bottleneck to alleviate in blood glucose management, which is currently not well explained in the paper. Related to this question, the authors may also consider justifying why making meal announcements more convenient/automated is not a good alternative to handle the blood glucose management problem. It will also be interesting to see how the proposed method will behave when augmented with meal announcements since in reality, the proposed solution might not always be reliable and intervention options like meal announcements could potentially improve the robustness of the solution.
Miscellaneous:
in Section 3.1 of the glucoregulatory system model G, the carbohydrate input $c_t$ seems to be considered as an aspect of the action. Based on the understanding of the reviewer, such an action is a proxy to the meal announcement and is not considered as an input from the user for the deployed policy. For better clarity, the authors can consider reporting the formula of the deployed policy and explain how the quantity $c_t$ is related to this policy.