Bridging the Gap Between Value and Policy Based Reinforcement Learning
We establish a new connection between value and policy based reinforcement learning (RL) based on a relationship between softmax temporal value consistency and policy optimality under entropy regularization. Specifically, we show that softmax consistent action values correspond to optimal entropy regularized policy probabilities along any action sequence, regardless of provenance. From this observation, we develop a new RL algorithm, Path Consistency Learning (PCL), that minimizes a notion of soft consistency error along multi-step action sequences extracted from both on-and off-policy traces. We examine the behavior of PCL in different scenarios and show that PCL can be interpreted as generalizing both actor-critic and Q-learning algorithms. We subsequently deepen the relationship by showing how a single model can be used to represent both a policy and the corresponding softmax state values, eliminating the need for a separate critic. The experimental evaluation demonstrates that PCL significantly outperforms strong actor-critic and Q-learning baselines across several benchmarks. 
By contrast, value based methods, such as Q-learning [44,22,30,42,21], can learn from any trajectory sampled from the same environment. Such "off-policy" methods are able to exploit data from other sources, such as experts, making them inherently more sample efficient than on-policy methods [10]. Their key drawback is that off-policy learning does not stably interact with function approximation [35,Chap.11]. The practical consequence is that extensive hyperparameter tuning can be required to obtain stable behavior. Despite practical success [22], there is also little theoretical understanding of how deep Q-learning might obtain near-optimal objective values.
Ideally, one would like to combine the unbiasedness and stability of on-policy training with the data efficiency of off-policy approaches. This desire has motivated substantial recent work on off-policy actor-critic methods, where the data efficiency of policy gradient is improved by training an offpolicy critic [19,21,10]. Although such methods have demonstrated improvements over on-policy actor-critic approaches, they have not resolved the theoretical difficulty associated with off-policy learning under function approximation. Hence, current methods remain potentially unstable and require specialized algorithmic and theoretical development as well as delicate tuning to be effective in practice [10,41,8].
In this paper, we exploit a relationship between policy optimization under entropy regularization and softmax value consistency to obtain a new form of stable off-policy learning. Even though entropy regularized policy optimization is a well studied topic in RL [46,39,40,47,5,4,6,7]-in fact, one that has been attracting renewed interest from concurrent work [25,11]-we contribute new observations to this study that are essential for the methods we propose: first, we identify a strong form of path consistency that relates optimal policy probabilities under entropy regularization to softmax consistent state values for any action sequence; second, we use this result to formulate a novel optimization objective that allows for a stable form of off-policy actor-critic learning; finally, we observe that under this objective the actor and critic can be unified in a single model that coherently fulfills both roles.
SUMMARY:
The paper considers entropy regularized discounted Markov Decision Process (MDP), and shows the relation between the optimal value, action-value, and policy. Moreover, it shows that the optimal value function and policy satisfy a temporal consistency in the form of Bellman-like equation (Theorem 1), which can also be extended to its n-step version (Corollary 2).
The paper introduces Path Consistent Learning by enforcing the temporal consistency, which is essentially a Bellman residual minimization procedure (Section 5).
SUMMARY OF EVALUATION:
Quality: Parts of the paper is sound (Section 3 and 4); parts of it is not (Section 5)
Clarity: The paper is well-written.
Originality: Some results seem to be novel, but similar ideas and analysis have been proposed/done before.
Significance: It might become an important paper in the entropy regularized RL literature.
EVALUATION:
This is a well-written paper that has some interesting results regarding the relation between V, Q, and pi for the entropy regularized MDP setting. That being said, I have some concerns, which I hope the authors clarify.
- Enforcing temporal consistency by minimizing its l2 error is minimizing the empirical Bellman error. Why do we need a new terminology (Path consistency) for this?
- The empirical Bellman error, either in the standard form or in the new form in this paper, is a biased estimate of the true Bellman error. Refer to the discussion in Section 3 of
Antos, Szepesvari, Munos, “Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path,” Machine Learning Journal, 2008.
Therefore, for stochastic dynamics, minimizing Equation (15) does not lead to the desired result. This is my most crucial comment, and I hope the authors provide an answer to it.
- Claiming that this paper is bridging the gap between the value and policy based RL is beyond what has actually been shown. The paper shows the relation between value and policy for a certain type of reward function (which has the entropy of policy as an additional reward term).
- On LL176-177, it is claimed that a new actor-critic paradigm is presented where the policy is not distinct from the values. Isn’t this exactly the same thing in the conventional value-based approach for which the optimal policy has a certain relationship with the optimal value function? In the standard setting, the greedy policy is optimal; here we have relationship of Equation (10).
- It might be noted that relations similar to Equations (7), (8), (9), and (10) have been observed in the context of Maximum Entropy Inverse Optimal Control. See Corollary 2 and the discussion after that in the following paper:
Ziebart, et al., “The Principle of Maximum Causal Entropy for Estimating Interacting Processes,” IEEE Transactions on Information Theory, 2013.
Here the Bellman equations are very similar to what we have in this paper.
Ziebart et al.’s framework is extended by the following paper to address problems with large state spaces:
Huang, et al., “Approximate MaxEnt Inverse Optimal Control and its Application for Mental Simulation of Human Interactions,” AAAI, 2016.
This latter paper concerns with the inverse optimal control (i.e., RL), but the approximate value iteration algorithms used for solving the IOC essentially use the same Bellman equations as we have here. The paper might want to compare the difference between these two approaches.
- One class of methods that can bridge the gap between value and policy based RL algorithms is classification-based RL algorithms. The paper ignores these approaches in its discussion.