State Aware Imitation Learning
Imitation learning is the study of learning how to act given a set of demonstrations provided by a human expert. It is intuitively apparent that learning to take optimal actions is a simpler undertaking in situations that are similar to the ones shown by the teacher. However, imitation learning approaches do not tend to use this insight directly. In this paper, we introduce State Aware Imitation Learning (SAIL), an imitation learning algorithm that allows an agent to learn how to remain in states where it can confidently take the correct action and how to recover if it is lead astray. Key to this algorithm is a gradient learned using a temporal difference update rule which leads the agent to prefer states similar to the demonstrated states. We show that estimating a linear approximation of this gradient yields similar theoretical guarantees to online temporal difference learning approaches and empirically show that SAIL can effectively be used for imitation learning in continuous domains with non-linear function approximators used for both the policy representation and the gradient estimate.
I've read the feedback, thanks for the clarifications.
%%%
This paper deals with imitation learning. It is proposed to maximize the posterior (policy parameters given the expert state-action pairs). The difference with a classic supervised learning approach is that here the fact that the policy acts on a dynamical system is taken into account, which adds the likelihood of states given policy (the states should be sampled according to the stationary distribution induced by the policy). This additional term is addressed through a temporal difference approach inspired by the work of Morimusa et al. [12], other terms are handled in the classic way (as in supervised learning). The proposed approach is experimented on two domains, a small on and a much larger one, and compared mainly to the supervised baseline (ignoring the dynamics).
This paper proposes an interesting, sound and well motivated contribution to imitation learning. I found the paper rather clear, but I think it could be clarified for readers less familiar with the domain. For example:
* in Eq. 3, the reverse probability appears. It would help to at least mention it in the text (it is not that classical)
* the constraint (l.168, expected gradient null) comes simply from the fact that the probabilities should sum to one. It would help a lot to say it (instead of just giving the constraint)
* the function f_w is multi-output (as many components as policy parameters), it could help to highlight this in the text
* the off-policy approach should be made explicit (notably, as it is a reversed TD approach, due to the reversed probabilities behind, do the importance weights apply to the same terms?)
Section 2 is rather complete, but some papers are missing. There is a line of work that tries to do imitation as a supervised approach while taking into account the dynamics (which is also what is done in this paper, the dynamics inducing a stationary distribution for each policy), for example :
* "Learning from Demonstration Using MDP Induced Metrics" by Melo and Lopes (ECML 2010)
* "Boosted and Reward-regularized Classification for Apprenticeship Learning" by Piot et al. (AAMAS 2014)
The second one could be easily considered as a baseline (it is a supervised approach whit an additional regularization term taking into account the dynamics, so the used supervised approach in the paper could be easily adapted)
Regarding the experiments:
* it would have been interesting to vary the number of episodes collected from the expert
* for the off-policy experiment, it would be good to compare also to (on-policy) SAIL, as a number of gathered trajectories
* for the walker, giving the number of input/output neurons would also help (28 in/4 out for the policy, 28 in/3200 out for the gradient estimator? so a 28:80:80:3200 network?) This makes big output networks, it could be discussed.
* still for the walker, what about off-policy learning?
Regarding appendix A, the gradient of the log-policy plays the role of the reward. Usually, rewards are assumed to be uniformly bounded, which may not be the case here (eg, with a linear softmax policy based on a tabular representation, the probabilities could be arbitrary close to zero, even if ever strictly positive). Isn't this a problem for the proof?