Learning to imitate expert actions given demonstrations containing image observations is a difficult problem in robotic control. The key challenge is generalizing behavior to out-of-distribution states that differ from those in the demonstrations. State-of-the-art imitation learning algorithms perform well in environments with low-dimensional observations, but typically involve adversarial optimization procedures, which can be difficult to use with high-dimensional image observations. We propose a remarkably simple alternative based on off-policy soft Q-learning, which we call soft Q imitation learning (SQIL, pronounced "skill"), that rewards the agent for matching demonstrated actions in demonstrated states. The key idea is initially filling the agent's experience replay buffer with demonstrations, where rewards are set to a positive constant, and setting rewards to zero in all additional experiences. We derive SQIL from first principles as a method for performing approximate inference under the MaxCausalEnt model of expert behavior. The approximate inference objective trades off between a pure behavioral cloning loss and a regularization term that incorporates information about state transitions via the soft Bellman error. Our experiments show that SQIL matches the state of the art in low-dimensional environments, and significantly outperforms prior work in playing video games from high-dimensional images.
The paper introduces a relatively simple method for imitation learning that seems to be successful despite its simplicity. The method, SQIL, assigns a constant positive reward (r) to the demonstrations and zero reward to generated trajectories. While I like the connections between SQIL and SQL and the simplicity of the idea, I think there are several issues which connections with GAIL that are not discussed; some "standard" environments (such as Mujoco) that SQIL has not compared against the baselines. I believe the paper would have a bigger impact after some addressing some of the issues.
(
Update: I am glad that the authors added updates to the experiments. I think the method could be practical due to the simplicity, therefore of interest to ICLR.
The Pong case is also quite interesting, although it seems slightly "unfair" since the true reward of Pong is also sparse and DQN could do well on it. I think the problem with GAIL is that the reward could be hard to learn in high-dimensional cases, so it is hard to find good hyperparameters for GAIL on the Pong case. This shows some potential of the idea behind using simple rewards.
)
1. The first issue is the similarity with GAIL in the "realistic" setting. Since we cannot have infinite expert demonstrations, there would always be some large enough network that could perfectly distinguish the demonstrations (assign reward to 1) and the generated policies (assign reward to 0). Therefore, it would seem to me that from this perspective SQIL is an instance of GAIL where the discriminator is powerful and expert demos are finite (and disjoint from generated trajectories, which is almost always the case for continuous control). In the finite capacity case, I am unsure whether the V and Q networks in SQIL does a similar job as the discriminator in GAIL / AIRL type algorithms, since both seem to extrapolate between demonstrations and generations?
2. Moreover, I don't think SQIL would always recover the expert policy even with infinite demonstrations. For example, lets think about the Reacher environment, where the agent controls a robotic arm to reach a target location. The expert demonstration is the fastest way of reaching the target (move minimum distance between joints). If we consider the MDP to have possibly very large / infinite horizon (only stops when it reaches the target), I could construct a hack policy that produces larger episodic reward compared to the expert. The policy would simply move back and forth between two expert demonstrated states, where it would receive 1 reward in the states for odd time and 0 reward for the states for even time. The reward would be something like 1 / (1 - \gamma^2) compared to the experts' reward which is \sum_{i=0..T} \gamma^{i} = (1 - \gamma^{T+1}) / (1 - \gamma). 
Some fix would be to set the reward for generated policies to be negative, or introduce some absorbing state where the expert will still receive the positive reward even after reaching the target (but that is not included in demonstrations). Nevertheless, a suitable reward prior seems to be crucial to the success of this SQIL, as with GAIL requiring reward augmentation.
3. Despite the above issues, I think this could be a very practical method due to its (perhaps surprising) simplicity compared to GAIL. However, the experiments only considered two environments that are not typically considered by GAIL; I believe SQIL would make a bigger impact if it is compared with GAIL in Mujoco environments -- seems not very implementation heavy because your code is based on OpenAI baselines anyway. Mujoco with image inputs would also be relevant (see ACKTR paper).
Minor points:
- What is the underlying RL algorithm for GAIL? It would seem weird if you use Q-learning for SQIL and TRPO for GAIL, which makes it impossible to identify whether Q-learning or SQIL contributed more to the performance. While GAIL used TRPO in the original paper, it would be relatively straightforward to come up with some version of GAIL that uses Q-learning. 
- Some more details in background for MaxEnt RL to make the paper more self contained.
- More details about the hyperparameters of SQIL in experiments -- e.g. what is \lambda?
- Did you pretrain the SQIL / GAIL policies? Either case, it would be important to mention that and be fair in terms of the comparison.
- Why does SQIL-11 perform worse than SQIL even though it is a tighter bound?
- wrt. math -- I think the anonymous comment addressed some of my concerns, but I have not read the updated version so cannot be sure.