Learning to imitate expert behavior from demonstrations can be challenging, especially in environments with high-dimensional, continuous observations and unknown dynamics. Supervised learning methods based on behavioral cloning (BC) suffer from distribution shift: because the agent greedily imitates demonstrated actions, it can drift away from demonstrated states due to error accumulation. Recent methods based on reinforcement learning (RL), such as inverse RL and generative adversarial imitation learning (GAIL), overcome this issue by training an RL agent to match the demonstrations over a long horizon. Since the true reward function for the task is unknown, these methods learn a reward function from the demonstrations, often using complex and brittle approximation techniques that involve adversarial training. We propose a simple alternative that still uses RL, but does not require learning a reward function. The key idea is to provide the agent with an incentive to match the demonstrations over a long horizon, by encouraging it to return to demonstrated states upon encountering new, out-of-distribution states. We accomplish this by giving the agent a constant reward of r = +1 for matching the demonstrated action in a demonstrated state, and a constant reward of r = 0 for all other behavior. Our method, which we call soft Q imitation learning (SQIL), can be implemented with a handful of minor modifications to any standard Q-learning or off-policy actor-critic algorithm. Theoretically, we show that SQIL can be interpreted as a regularized variant of BC that uses a sparsity prior to encourage long-horizon imitation. Empirically, we show that SQIL outperforms BC and achieves competitive results compared to GAIL, on a variety of image-based and low-dimensional tasks in Box2D, Atari, and MuJoCo. This paper is a proof of concept that illustrates how a simple imitation method based on RL with constant rewards can be as effective as more complex methods that use learned rewards.
Summary
-------------
The authors propose SQUIL, an off-policy imitation learning (IL) algorithm which attempts to overcome the classic drift problems of behavioral cloning (BC). The idea is to reduce IL to a standard RL problem with a reward that incentivizes the agent to take expert actions in states observed in the demonstrations. The algorithm is tested on both image-based tasks (Atari) and continuous control tasks (Mujoco) and shown effective against GAIL and a simple supervised BC approach.
Comments
--------
Overcoming the limitations of BC algorithms is a relevant problem and this work presents a simple and interesting solution. The idea is to use any off-policy RL algorithm (assuming access to the true environment) with a reward that favors imitation. The paper is well-organized and easy to read. The experiments are also quite convincing and demonstrate that the proposed methodology is, in fact, more robust to distribution shift (i.e., less prone to overfitting the train set) than standard supervised BC and GAIL. Some comments/questions follow.
1. The considered settings (e.g., MDPs, IL/IRL problems) are never formalized in the paper (which directly starts by describing the method in Section 2). It would be good to add some preliminaries to allow even readers unfamiliar with this problem to easily go through the paper.
2. Though the approach solves one of the main problems of many BC/non-IRL algorithms, it still has some of the other limitations. For instance, the recovered policy is non-transferable to different dynamics, while a reward function is. Furthermore, it cannot recover optimal behavior if the expert is sub-optimal.
3. In many practical cases, the assumption that we can access the true environment is not reasonable. Suppose, for instance, that we have a dataset of driving demonstrations from a real car, and that is all we can obtain. Of course, we cannot start interacting with the real car to imitate the driver's behavior and we need to rely on simulators, which poses additional difficulties due to the inevitable approximations. Do the authors think that a batch RL algorithm could be adopted to solve this task (i.e., by solely using the expert data)? The ablation study in section 4 seems to answer this question negatively, at least for large distribution shifts.
4. Below equation (1), why "r does not depend on the state or action"? It should be R(s,a) = INDICATOR{(s,a) \in D_{demo}}, so it does depend on (s,a).
5. D_{demo} is missing from the inputs to Algorithm 1 
6. I did not understand why, in Section 3.2 (above Equation 8), the authors mention that (7) squared is equivalent to the squared Bellman error of Equation 1. Where is r? It should be equivalent only from (s,a) \in D_samp, for which r=0, but for (s,a) \in D_demo we have r=1 and that term is missing in (7). Have I misunderstood something?
7. In Figure 1, why does GAIL perform better with shift than no-shift?
8. Why are some curves in Figure 3 (bottom) above the expert performance?
9. It would be interesting to show how the proposed method compares to IRL-based approaches for continuous states/actions such as [1] or [2]. These algorithms should be applicable to the considered domains (at least Mujoco ones).
[1] Finn, C., Levine, S., & Abbeel, P. (2016, June). Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning (pp. 49-58).
[2] Boularias, A., Kober, J., & Peters, J. (2011, June). Relative entropy inverse reinforcement learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (pp. 182-189).