Reinforcement learning (RL) offers a principled way to achieve the optimal cumulative performance index in discrete-time nonlinear stochastic systems, which are modeled as Markov decision processes. Its integration with deep learning techniques has promoted the field of deep RL with an impressive performance in complicated continuous control tasks. However, from a control-theoretic perspective, the first and most important property of a system to be guaranteed is stability. Unfortunately, stability is rarely assured in RL and remains an open question. In this paper, we propose a stability guaranteed RL framework which simultaneously learns a Lyapunov function along with the controller or policy, both of which are parameterized by deep neural networks, by borrowing the concept of Lyapunov function from control theory. Our framework can not only offer comparable or superior control performance over the state-of-the-art RL algorithms, but also construct a Lyapunov function to validate the closed-loop stability. In the simulated experiments, our approach is evaluated on several well-known examples including classic CartPole balancing, 3-dimensional robot control and control of synthetic biology gene regulatory networks. Compared with RL algorithms without stability guarantee, our approach can enable the system to recover to the operating point when interfered by uncertainties such as unseen disturbances and system parametric variations to a certain extent. (Anonymous code is available to reproduce the experimental results 1 .)
######## Rebuttal Response:
Thanks for the thorough response.
Q2: The title still hasn’t changed on the current draft
Q4: To be more precise: 
‘a novel data-based approach for analyzing the stability of the closed-loop system is proposed by constructing a Lyapunov function parameterized by deep neural network’ - this alone is not novel, you would need to specify how your method of doing this is new 
‘a practical learning algorithm is designed to search the stability guaranteed controller’ - this is a natural consequence of contribution 1, some further justification is needed as to why this could be viewed as an interesting contribution (i.e. the Lagrangian approach, if this is novel)
‘ the learned controller is able to stabilize the system when interfered by uncertainties such as unseen disturbance and system parameters variations of certain extent’ - this is not a contribution, but an experimental result.
Q5: The review believes that model-free control and stability-guarantees are fundamentally orthogonal ideas, rather than just under-studied work as the authors have been suggesting in the script and rebuttal. Given that discrete-time Lyapunov stability is defined through expressions along the lines of L(f(x)) - L(x) < 0, for Lyapunov function L and closed-loop dynamics f, claiming that stability is being ‘analyzed’ without f is disingenuous. Instead, by making the value function a Lyapunov function, the goal is that the *converged* value function should produce a stable policy, and still, this is surely only assured within the space of samples. Moreover, the use of the discount factor \gamma, popular in MFRL, essentially acts as a time horizon, so I’m not convinced a Lyapunov function learned with a \gamma < 1 can be called stable in the pure infinite-horizon sense.
With this in mind, I think the work would benefit from a revised central claim: that the use of Lyapunov value functions (as an inductive bias) provides more *robust* model-free controllers. I believe this message highlights the value of this work for MFRL, without making false assertions. This, in particular, would highlight the fact that many MFRL algorithms are benchmarked on deterministic environments, and therefore incredible brittle as the experimental results suggest.
Q9: This remark was aimed at earlier in the paper, either the introduction or main section, rather than the experimental section. The fact that a value function can be viewed as a Lyapunov function makes sense but I’m not sure it is a well-known fact in the wider community. Basically, an introduction to the intersection of Lyapunov stability and optimal control would improve the paper.
Q:10 The fact that the clipping of the multiplier corresponds to unstable policies during learning demonstrates that this pitfall needs to be expressed explicitly. Whether the stability guarantees apply to the converged policy or also intermediate policies is not clear on the initial reading of the paper.
For me, this highlights another weakness in the paper. This initial theorems talk of L(s), which relates to the critic L_c by L(s) = E_{u\sim\pi(s)} [L_c(s,u)], however in the subsequent objectives (eg Eq 2), this marginalization never occurs, therefore I don’t feel like you can say Theorem 2 applies to your resultant algorithm.  Moreover, with the \alpha_3 c term in Equation 3, c should be c(s, a) with a marginalized, which it doesn’t appear to be, and the hyperparameter alpha_3 is never discussed nor tuning explained. Assuming this not done in the code, the experiments need to be re-evaluated with Theorem 2 properly enforced through a sample approximation of the marginalization.
Q11/Q12: Thank you for the Markov jump experiments. I’m not sure I understand why LAC is able to learn the task while SAC cannot. To me, this suggests perhaps a lack of hyperparameter tuning for SAC or further investigation. Moreover, there are typos in captions Fig 1, e and f.
A note of figures: Please ensure all axes should be labeled and should be of sufficient size. Many are too small and unreadable. Figure 2 looks like it could be 1 plot (though perhaps requires normalization).
Given that the strength of this method is the added robustness upon convergence, I think it would be valuable to focus less on time-domain results (Figure 3) (these can be added to the appendix for clarity), but instead show how each parameter/noise variation affects the mean and variance of the episodic return. I would expect that, while LAC provides significant robustness, it is still limited. The results don’t demonstrate this. It would also be interesting to know which hyperparameter controls this limit. I imagine there is a robustness/performance tradeoff.  
In conclusion, while I appreciate the efforts the authors put into the rebuttal, the extended discussions made me rethink my rating and I have decreased my rating to reject. I believe fixing the issues highlighted above and redrafting the central message must be done before this paper is ready for publication. 
######## Review:
This paper investigates the use of Lyanpunov theory as an inductive bias for improving the stability / robustness of policies in a model-free actor-critic reinforcement learning setting. Through viewing the Critic as a Lyapunov function, optimizing the policy with a Lyapunov-based constraint is meant to ensure the stability of the policy through a ‘cost stability’ metric.. Experimental results show that Lyapunov-based Soft-Actor Critic (LAC) is more robust than SAC on some linear and nonlinear environments.
The reviewer believes that the study of intersections between Control Theory and RL to be immensely valuable and the authors outline a principled formulation. However,  the implementation, experiments and general manuscript suggest that paper requires further work before it is conference-ready. 
As the author understands it, the current state of the literature of Lyapunov methods for Deep Reinforcement Learning can be summarized as:
Richards et al, 2018, Classify stable region and learn neural Lyapunov function for a safe exploration strategy
Berkenkamp et al, 2018: Classify the stable region via GP, move there for exploration
Chow et al 2018 Constrained MDPs for discrete gridworld environments
Chow et al 2019 Constrained MDPs for continuous environments through a projection on the policy
This work: Actor-Critic constrained policy optimization with a lyapunov-based value function critic
In the introduction and the related work, too much emphasis is put on explaining stability and discussing methods like Model Predictive Control (MPC) which do not benefit the rest of the paper. Additionally, the three contributions listed do not seem particularly novel given the past literature. 
The premise of the formulation also presents several unquestioned assumptions and design decisions:
Why model-free RL, as the authors also state that many samples are required to validate stability?
How does the requirement of stability inform the search strategy in this work? Especially as SAC uses a maximum entropy stochastic policy to aid exploration. 
Do you really get ‘guarantees’ with sample-based methods? I would expect bounds based on the number of samples
The cost-based measure of stability seems open to abuse - i.e. for the half-cheetah environment only the centre-of-mass horizontal velocity in covered in the cost function, the stability of the embodiment (joint angles and velocities) are ignored. For the Fetch Reacher, a cost function in cartesian space ignores instabilities from kinematic singularities in joint space. I would image the cost function needs to be a measure on the entire dynamic state. 
The notion of a Value function as a Lyapunov function is very interesting, and since it was the basis of the work, would have benefitted from more discussion, i.e. for which cost/reward function families the equivalence is valid for, and how it compares to other Lyapunov candidate functions.
With the RL formulation, the requirement of clipping with the lagrangians is suspicious, as it suggests the objective and/or its numerics are not well posed.
With the choice of experiments, they do not seem to question the central problem outlined by the paper. Rather than show environments SAC returns unstable trajectories during learning, the experiments aim to demonstrate instead a general robustness. The reviewer appreciates that stability is difficult to assess; however, while stability is heavily linked to robustness, a paper title promising stability guarantees should demonstrate some strong empirical evidence stability.
Additionally, the choice of environments do not seem to be ideal test beds for stability - i.e, the half-cheetah is stabilized via interactions with the ground. The reviewer would prefer to see simpler nonlinear environments, such as Markov Jump Processes / Switching Linear Dynamics, where SAC clearly demonstrates instability during learning which LAC is sufficiently regularized against. Additionally, while the `repressilator’ is an interesting application to the domain of bioengineering, its addition does not seem to be especially motivated by the central goal of the paper, so just adds to confuse the reader with unnecessary theoretical content.
Moreover, a brief literature review uncovered some relevant earlier work which was not cited: 
Construction of neural network based Lyapunov functions, Petridis et al, 2006
Generation of Lyapunov functions by neural networks, Noroozi et al, 2008
Lyapunov Design for Safe Reinforcement Learning, Perkins et al, 2002
Some of the references also appear incorrectly formatted or incorrect, i.e. the reference for Spencer et al, 2018 should be the CoRL 2018 version rather than arxiv. 
Also, the general use of grammar in the manuscript would benefit from another draft. In particular, the title could be improved, i.e.
    Model-free Control of Nonlinear Stochastic Systems with Stability Guarantees