A Lyapunov-based Approach to Safe Reinforcement Learning
In many real-world reinforcement learning (RL) problems, besides optimizing the main objective function, an agent must concurrently avoid violating a number of constraints. In particular, besides optimizing performance, it is crucial to guarantee the safety of an agent during training as well as deployment (e.g., a robot should avoid taking actions -exploratory or not -which irrevocably harm its hardware). To incorporate safety in RL, we derive algorithms under the framework of constrained Markov decision processes (CMDPs), an extension of the standard Markov decision processes (MDPs) augmented with constraints on expected cumulative costs. Our approach hinges on a novel Lyapunov method. We define and present a method for constructing Lyapunov functions, which provide an effective way to guarantee the global safety of a behavior policy during training via a set of local linear constraints. Leveraging these theoretical underpinnings, we show how to use the Lyapunov approach to systematically transform dynamic programming (DP) and RL algorithms into their safe counterparts. To illustrate their effectiveness, we evaluate these algorithms in several CMDP planning and decision-making tasks on a safety benchmark domain. Our results show that our proposed method significantly outperforms existing baselines in balancing constraint satisfaction and performance.
The paper focuses on the problem of safety in RL. The safety concept is expressed through a constraint on a cumulative cost. The authors leverage the framework of Constrained MDPs (CMDPs) to derive a novel approach where the Lyapunov function can be automatically constructed (instead of being manually defined). The core contribution is a method for constructing the Lyapunov function that guarantees the safety of the policy during training. In particular, safety can be efficiently expressed through a set of linear constraints. Leveraging this method, the authors showed how to modify several dynamic programming approaches to enforce safety. These algorithms do not have guarantees of convergence but the authors have empirically shown that they often learn near-optimal policies.  Comments ----------- The idea proposed in the article is very interesting. In particular, I believe that safety is an important topic to be explored in RL with a potentially high impact in real-world applications.  The introduction is well written and provides a nice overview of the literature about CMDPs.  On contrary, I believe that preliminaries can be made clearer. I think that it is necessary to state clearly the settings. It took me a while to figure it out that the considered scenario is the stochastic shortest path problem which is an instance of total cost problem. In particular, I was surprised by the fact that the Bellman (policy) operator is a contraction mapping (that is not in general true in undiscounted settings). After checking [Bertsekas, 1995] I realized that T is a contraction since, in your problem,  all stationary policies are proper [Bertsekas, 1995, Chap 2, Def 1.1, pag 80], that is, when termination is inevitable under all stationary policies. I suggest you to clearly state the settings and to put explicit references, e.g., in line 134 you should put page and prop/lem when references [7]. Am I correct? Moreover, while checking the proof in Sec C.3, I was not able to find the reference to Prop 3.3.1 and Prop 1.5.2 in [7]. Can you please provide page and or chapter? Please, also update the reference. Is it the volume 2?  I believe that the analysis can be generalized outside this settings. However, you should rely on different arguments since the Bellman operator is no more a contraction in generic undiscounted problems. Could you please comment on this? Is it possible to generalize to average cost problems?  Apart from that, I think that the presentation is fair and you acknowledge correctly strengths and weaknesses of the approach (e.g., the fact that algorithms do not enjoy optimality guarantees).  I was surprised by the fact that there is no comparison with safe methods that guarantee a monotonic policy performance improvement: e.g., in policy iteration [Kakade and Langford, 2002; Pirotta et al, 2013a] or policy gradient [Pirotta et al, 2013b, etc.]. Your approach is definitely different but, often, the constraint cost can be incorporated in the reward (it the case of your example).  The mentioned methods are guaranteed to learn policies that are always better than the baseline one. Then, if the baseline policy is good enough (satisfies the constraints reformulated in terms of bad reward) these approaches are guaranteed to recover safe policy over the entire learning process.  I think you should mention these works in the paper highlighting similarities and advantages of your approach. Please comment on this.  Please add a reference for the fact that the first hitting time is upper bounded by a finite quantity (you talk about standard notion).  I overall like the paper and I think it will be a good fit for NIPS subject to the fact that the authors will take into account the aforementioned comments about the presentation.  - Line 706: fnction -> function - Policy Distillation algorithm in page 24 contains a broken reference    [Bertsekas, 1995] D. P. Bertsekas, Dynamic Programming and Optimal Control, Vol. II. Athena Scientific ISBN: 1-886529-13-2 [Pirotta et al. 2013a] M Pirotta, M Restelli, A Pecorino, D Calandriello. Safe Policy Iteration, ICML 2013 [Pirotta et al. 2013b] M Pirotta, M Restelli, L Bascetta. Adaptive step-size for policy gradient methods, NIPS 2013  ------------------------ After feedback Thank you very much for the feedback. Now it is clear to me the difference between safe approaches and the proposed algorithm. I suggest you add this explanation to the paper. Hitting time: The assumption of boundness of the first hitting time is very restrictive. I underestimated it during the review because I was not sure about the meaning of the term "uniformly bounded". I checked the reference and uniform boundness of the first hitting time means that the time to reach the absorbing state is almost surely bounded for any stationary policy. I start noticing that due to the settings you considered, every stationary policy induces an absorbing Markov chain (by definition of proper policy, [Bertsekas, 1995, Chap 2, Def 1.1, pag 80]). For absorbing Markov chains it is known that the boundness of the first hitting time (with probability 1 the time to reach the absorbing state is bounded) implies that there are no loops in the chain (every state is visited at most once with probability 1) [see for example App. D of Fruit and Lazaric, Exploration-Exploitation in MDPs with Options, 2017]. You can assume this property but I would like to see this comment in the paper because I believe that the strong implication of this assumption is not well known in general. Moreover, you need to clearly distinguish the theoretical analysis from the empirical results because this assumption seems to be not satisfied in your experiments.