Reward Constrained Policy Optimization
Solving tasks in Reinforcement Learning is no easy feat. As the goal of the agent is to maximize the accumulated reward, it often learns to exploit loopholes and misspecifications in the reward signal resulting in unwanted behavior. While constraints may solve this issue, there is no closed form solution for general constraints. In this work, we present a novel multi-timescale approach for constrained policy optimization, called 'Reward Constrained Policy Optimization' (RCPO), which uses an alternative penalty signal to guide the policy towards a constraint satisfying one. We prove the convergence of our approach and provide empirical evidence of its ability to train constraint satisfying policies.
The paper introduces RCPO, a model-free deep RL algorithm for learning optimal policies that satisfy some per-state constraint on expectation. The derivation of the algorithm is quite straightforward, starts from the definition of constrained optimization problem, and proceed by forming and optimizing the Lagrangian. Additionally, a value function for the constraint is learned. The algorithm is only compared to a baseline optimizing the Lagrangian directly using Monte-Carlo sampling.
The paper has two major problems. First, while the derivation of the method makes intuitively sense, it is supported by vaguely stated theorems, which mixes rigorous guarantees with practical approximations. For example, Equation 4 assumes strong duality. How would the result change if weak duality was used instead? The main result in Theorem 1 makes the assumption that dual variable is constant with respect the policy, which might be true in practice, but it is not obvious how the approximation affects the theory. Further, instead of simply referring to prior convergence results, I would strongly suggest including the exact prior theorems and assumptions in the appendix.
The second problem is the empirical validation, which is incomplete and misleading. Constrained policy optimization is not a new topic (e.g. work by Achiam et al.), so it is important to compare to the prior works. It is stated in the paper that the prior methods cannot be used to handle mean value constraints. However, it would be important to include experiments that can be solved with prior methods too, for example the experiments in Achiam at al. for proper comparison. The results in Table 2 are confusing: what makes the bolded results better than the others? If the criterion is highest return and torque < 25%, then \lambda=0.1 should be chosen for Hopper-v2. Also, The results seem to have high variance, and judging based on Table 2 and Figure 3, it is not obvious how well RCPO actually works.
To summarize, while the topic is undoubtedly important, the paper would need be improved in terms of better differentiating the theory from practice, and by including a rigorous comparison to prior work.
Minor points:
- What is exactly the difference between discounted sum constraint and mean value constraint?
- Could consider use colors in Table 1.
- Section 4.1.: What does “... enables training using a finite number of samples” exactly mean in this case?
- Table 2: The grid for \lambda is too sparse. 
- Proof of Theorem 1: What does it mean \theta to be stable?
- Proof of Theorem 2: “Theorem C” -> “Theorem 1”