In many reinforcement learning applications, it is crucial that the agent interacts with the environment only through safe policies, i.e., policies that do not take the agent to certain undesirable situations. These problems are often formulated as a constrained Markov decision process (CMDP) in which the agent's goal is to optimize its main objective while not violating a number of safety constraints. In this paper, we propose safe policy optimization algorithms that are based on the Lyapunov approach to CMDPs, an approach that has well-established theoretical guarantees in control engineering. We first show how to generate a set of state-dependent Lyapunov constraints from the original CMDP safety constraints. We then propose safe policy gradient (PG) algorithms that train a neural network policy using deep deterministic policy gradient (DDPG) or proximal policy optimization (PPO), while guaranteeing near-constraint satisfaction at every policy update by projecting either the policy parameter or the action onto the set of feasible solutions induced by the linearized Lyapunov constraints. Unlike the existing (safe) constrained policy gradient algorithms, ours are more data efficient as they are able to utilize both on-policy and off-policy data. Furthermore, the action-projection version of our algorithms often leads to less conservative policy updates and allows for natural integration into an end-to-end PG training pipeline. We evaluate our algorithms and compare them with constrained policy optimization (CPO) and the Lagrangian method on several high-dimensional continuous state and action simulated robot locomotion tasks, in which the agent must satisfy certain safety constraints while minimizing its expected cumulative cost.
￼
To my understanding, this paper builds on prior work from Chow et al. to apply Lyapunov-based safe optimization to the policy-gradient setting. This seems is similar to work by Achiam 2017. While this work seems like an interesting framework for encompassing several classes of constrained policy optimization settings in the Lyapunov-based setting, I have some concerns about the evaluation methodology. 
It is claimed that the paper compares against “two baselines, CPO and the Lagrangian method, on several robot locomotion tasks, in which the agent must satisfy certain safety constraints while minimizing its expected cumulative cost.” Then it is stated in the experimental section “However since backtracking line-search in TRPO can be computationally expensive, and it may lead to conservative policy updates, without loss of generality we adopt the original construction of CPO to create a PPO counterpart of CPO (which coincides with SPPO) and use that as our baseline.” This seems directly to contrast to the earlier statement which states that it is unclear how to modify the CPO methodology to other RL algorithms. Moreover, is this really a fair comparison? The original method has been modified to form a new baseline and I’m not sure that it is “without loss of generality”. 
Also, it is unclear whether the results can be accepted at face value. Are these averaged across several random seeds and trials? Will performance hold across them? What would be the variance? Recent work has shown that taking 1 run especially in MuJoCo environments doesn’t necessarily provide statistically significant values. In fact the original CPO paper shows the standard deviations across several random seeds and compares directly against an earlier work in this way (PDO). Moreover, it is unclear why CPO was not directly compared against and neither was the "equivalent" baseline not compared on similar environments as in the original CPO paper. 
Comments:
Figure 3 is difficult to parse, the ends of the graphs are cut off. Maybe putting the y axis into log format would help with readability here or having the metrics be in a table.