Robust real-world learning should benefit from both demonstrations and interaction with the environment. Current approaches to learning from demonstration and reward perform supervised learning on expert demonstration data and use reinforcement learning to further improve performance based on reward from the environment. These tasks have divergent losses which are difficult to jointly optimize; further, such methods can be very sensitive to noisy demonstrations. We propose a unified reinforcement learning algorithm, Normalized Actor-Critic (NAC), that effectively normalizes the Q-function, reducing the Q-values of actions unseen in the demonstration data. NAC learns an initial policy network from demonstration and refines the policy in a real environment. Crucially, both learning from demonstration and interactive refinement use exactly the same objective, unlike prior approaches that combine distinct supervised and reinforcement losses. This makes NAC robust to suboptimal demonstration data, since the method is not forced to mimic all of the examples in the dataset. We show that our unified reinforcement learning algorithm can learn robustly and outperform existing baselines when evaluated on several realistic driving games.
SUMMARY:
The motivation for this work is to have an RL algorithm that can use imperfect demonstrations to accelerate learning. The paper proposes an actor-critic algorithm, called Normalized Actor-Critic (NAC), based on the entropy-regularized formulation of RL, which is defined by adding the entropy of the policy as an additional term in the reward function.
Entropy-regularized formulation leads to nice relationships between the value function and the policy, and has been explored recently by many, including [Ziebart, 2010], [Schulman, 2017], [Nachum, 2017], and [Haarnoja, 2017].
The paper benefits from such a relationship and derives an actor-critic algorithm. Specifically, the paper only parametrizes the Q function, and computes the policy gradient using the relation between the policy and Q function (Appendix A.1).
Through a set of experiments, the paper shows the effectiveness of the method.
EVALUATION:
I think exploring and understanding entropy-regularized RL algorithm is important. It is also important to be able to benefit from off-policy data. I also find the empirical results encouraging. But I have some concerns about this paper:
- The derivations of the paper are unclear.
- The relation with other recent work in entropy-regularized RL should be expanded.
- The work is less about benefiting from demonstration data and more about using off-policy data.
- The algorithm that performs well is not the one that was actually derived.
* Unclear derivations:
The derivations of Appendix A.1 is unclear. It makes it difficult to verify the derivations.
To begin with, what is the loss function of which (9) and (10) are its gradients?
To be more specific, the choices of \hat{Q} in (15) and \hat{V} in (19) are not clear.  For example, just after (18) it is said that “\hat{Q} could be obtained through bootstrapping by R + gamma V_Q”. But if it is the case, shouldn’t we have a gradient of Q in (15) too? (or show that it can be ignored?)
It appears that \hat{Q} and \hat{V} are parameterized independently from Q (which is a function of theta). Later in the paper they are estimated using a target network, but this is not specified in the derivations.
The main problem boils down to the fact that the paper does not start from a loss function and compute all the gradients in a systematic way. Instead it starts from gradient terms, each of which seems to be from different papers, and then simplifies them. For example, the policy gradient in (8), which is further decomposed in Appendix A.1 as (15) and (16) and simplified, appears to be Eq. (50) of [Schulman et al., 2017] (https://arxiv.org/abs/1704.06440). In that paper we have Q_pi instead of \hat{Q} though.
I suggest that the authors start from a loss function and clearly derive all necessary steps.
* Unclear relation with other papers:
What part of the derivations of this work are novel? Currently the novelty is not obvious.
For example, having the gradient of both Q and V, as in (9), has been stated by [Haarnoja et al., 2017] (very similar formulation is developed in Appendix B of https://arxiv.org/abs/1702.08165).
An algorithm that can work with off-policy data has also been developed by [Nachum, 2017] (in the form of a Bellman residual minimization algorithm, as opposed to this work which essentially uses a Fitted Q-Iteration algorithm as the critic).
I think the paper could do a better job differentiating from those other papers.
* The claim that this paper is about learning from demonstration is a bit questionable. The paper essentially introduces a method to use off-policy data, which is of course important, but does not cover the important scenario where we only have access to (state,action) pairs given by an expert. Here it appears from the description of Algorithm 1 that the transitions in the demonstration data have the same semantic as the interaction data, i.e., (s,a,r,s’).
This makes it different from the work by [Kim et al., 2013], [Piot et al., 2014], and [Chemali et al., 2015], which do not require such a restriction on the demonstration data.
* The paper mentions that to formalize the method as a policy gradient one, importance sampling should be used (the paragraph after (12)), but the performance of such a formulation is bad, as depicted in Figure 2. As a result, Algorithm 1 does not use importance sampling.
This basically suggests that by ignoring the fact that the data is collected off-policy, and treating it as an on-policy data, the agent might perform better. This is an interesting phenomenon and deservers further study, as currently doing the “wrong” things is better than doing the “right” thing. I think a good paper should investigate this fact more.