Exponentially Weighted Imitation Learning for Batched Historical Data
We consider deep policy learning with only batched historical trajectories. The main challenge of this problem is that the learner no longer has a simulator or "environment oracle" as in most reinforcement learning settings. To solve this problem, we propose a monotonic advantage reweighted imitation learning strategy that is applicable to problems with complex nonlinear function approximation and works well with hybrid (discrete and continuous) action space. The method does not rely on the knowledge of the behavior policy, thus can be used to learn from data generated by an unknown policy. Under mild conditions, our algorithm, though surprisingly simple, has a policy improvement bound and outperforms most competing methods empirically. Thorough numerical results are also provided to demonstrate the efficacy of the proposed methodology.
Summary: The paper considers imitation learning with only batched historical trajectories and without any further access to any simulators or environment models. The proposed imitation learning is reward-aware, in a sense that it weights the expert's action based on advantage of the expert policy A^{\pi}. The paper also analyze the proposed algorithm and show a policy improvement over the expert policy.    Strengths: The setting considered in the paper is challenging: the learning algorithm only has access to batched trajectories from the expert but further access to the simulator or model. Hence most of previous imitation learning works (e.g., Dagger ,GAIL) won't apply in this setting. Simply Behavior cloning will work in this setting, but the paper shows that by leveraging the advantage of the expert, it can outperform Behavior cloning.    Main comments below:  1. The definition and the use of KL divergence between two policies, i.e., D_{KL}(\pi'||pi) is not clear. Specifically, the definition in Eq (1) is different from the definition of KL below line 56. Which definition did you use in the rest of the paper, say, in Eq (3)?   2. Regarding the state distribution rho(s), I assume it is always referring to the state distribution of the behavior policy pi. However, below line 146, when you say that it is imitating a new hypothetic policy \tilde{\pi}, shouldn't the state distribution becomes the state-distribution resulting from \tilde{\pi}? The state distribution of tilde{\pi} and \pi could be different.   3. Line 147: Why C(s) could be emitted? It is state-dependent. Ignoring C(s) could affect the solution of the optimization. Could the authors elaborate on why it is ok to ignore C(s)?   4. Even assume that we have a rich enough policy space such that we can drive D_{KL}(\tilde{\pi} || \pi_{\theta}) to zero. How does this guarantee that \pi_{\theta} is also as good as, or even better than pi? Note that D_{KL}(\tilde{\pi}|| \pi) = 0 does not imply that \tilde{\pi} and \pi are the same under every state: it depends on whether or not state distribution rho(s) has non-zero probability on every state (consider the case where. a behavior policy can only cover a small part of the large state space). Also no matter how deeper the network is, we can never ever make D_{KL}(\tilde{\pi}|| \pi_{\theta}) reach zero, simply because we do not have infinitely many data. Assuming, for instance, D_{KL}(\tilde{\pi} || pi_{\theta}) <= epsilon for some small epsilon, is reasonable, but then we need to analyze how epsilon affect the performance of the learned \pi_{\theta}.  5. Notations in Eq (7) - (10) are not well-defined. Could you point the readers to the places where, for instance, E_{\pi} \log \pi_{\theta} is defined? What does it even mean by log(\pi_{\theta}), as pi_{\theta} is a function? Also the KL used in (7) and (10) are not consistent. Following the definition in (7), shouldn't the expectation in (10) be defined with respect to \tilde{\pi}?     After rebuttal:   Thanks a lot for the rebuttal. I read all reviews and the rebuttal. Unfortunately I'm not going to raise my score.   Regarding the theory section, essentially the paper is trying to imitate an ideal policy (\tilde{\pi} in Eq 4) using function approximations (\pi_{\theta}). The paper claims the algorithm 1 essentially is minimizing D_{KL}(\tilde{\pi} || \pi_{theta}), but I'm not sure if it's a good idea. Again, the state distribution problem plays important role here. Note that in Alg 1, it is using states from the behavior policy, which is neither \tilde{\pi} nor \pi_{\theta}!. (Existing analysis of Behavior cloning assumes we generate state-action pairs from the policy that we are going to imitate! However here we do not have data from \tilde{\pi}---the policy that we are supposed to imitate).  To behavior clone this ideal policy, one should minimize KL under the state-distribution resulting from the ideal policy \tilde{\pi}. This is extremely important for the discussion in section 5.2. While it's not hard to see \tilde{\pi} is a uniformly better policy than the behavior policy \pi, it is definitely not clear to me that the learned policy pi_{\theta} would be better than or at least as good as \pi, considering that the state-distribution for measuring KL-divergence between \tilde{\pi} and \pi_{\theta} is the state-distribution of the behavior policy \pi.   Overall, I might be wrong but I personally think that the paper somehow implicitly kept using max_{s} KL(\pi(. | s) || \pi'(. | s) ) as the default-to-go definition of KL-divergence between two policies, at least throughout the analysis (if we can drive this kl-divergence to zero or small number epsilon, then pi and pi' will be close to each other for any states, hence in analysis we do not need to worry about the state-action distribution). But in reality, we can only measure KL under the state-distribution of behavior policy (i.e., max_{s} is impossible to evaluate in large state space). Theoretical guarantees of imitation learning where state-action distribution induced neither from the expert policy, i.e., \tilde{\pi} (classic Behavior cloning analysis considers this setting), nor from the learned policy, i.e. \pi_{\theta} (method like DAgger considers this setting) seem not trivial and need more careful analysis.