Generative Adversarial Imitation Learning
Consider learning a policy from example expert behavior, without interaction with the expert or access to a reinforcement signal. One approach is to recover the expert's cost function with inverse reinforcement learning, then extract a policy from that cost function with reinforcement learning. This approach is indirect and can be slow. We propose a new general framework for directly extracting a policy from data as if it were obtained by reinforcement learning following inverse reinforcement learning. We show that a certain instantiation of our framework draws an analogy between imitation learning and generative adversarial networks, from which we derive a model-free imitation learning algorithm that obtains significant performance gains over existing model-free methods in imitating complex behaviors in large, high-dimensional environments.
The paper is concerned with methods trying to imitate expert behavior via imitation learning. It presents both a theoretical investigation into connections between existing imitation learning (or more concretely, apprenticeship learning) algorithms and a new algorithm for imitation learning via a method akin to the training of generative adversarial networks. Extensive experiments are conducted on a variety of benchmarks showing the efficacy of the derived algorithm.
First off, this is a clearly written and well thought through paper that absolutely deserves to be published at NIPS.  The derivation of inverse reinforcement learning as the dual of learning an occupancy measure in the state-action space is clear, concise, and allows the authors to highlight connections between different, existing, IRL algorithms. These were interesting to discover for me as a reader and the proofs all seem sound.  The newly derived adversarial imitation learning method, which draws on a connection to generative adversarial networks (GANs) (formulating the problem of learning an occupancy measure as a classification problem), is interesting and seems less limited than the previously existing state of the art (in terms of applicability to a wide range of different tasks). I suspect that we will see several new advancements based on this new algorithm as it naturally invites for further application of existing deep learning machinery to the problem.  One argument that could be made against the paper is that it is fairly dense, packing a lot of material into 8 pages, and thus fails to be completely self-contained (i.e. the steps required to optimize the policy via TRPO are not well explained). While true, I believe this generally to be tolerable in this case as the paper is well written and explains all necessary concepts in enough detail for the reader to follow. However, I do believe that an explanation of the exact algorithm, including the TRPO steps would be useful for reproducing the paper, and aiding the interested reading in following each step. I thus would encourage the authors to add such a description to the supplementary material.  The only real issue I have with the paper is the fact that the authors seem to gloss over the problems that using an GAN like procedure can introduce. It is well known that GANs can be difficult to train and I suspect that the usage in the presented algorithm makes no exception to that rule. To that end it would be interesting to understand which (if any) countermeasures the authors took to stabilize training. Touching on the same subject, the objective maximized by the discriminator (Equation (19)) results in rewards going towards -inf for the policy (in cases where it does not currently match the expert behaviour), this could result in very large variances in the likelihood ratio estimator used for optimizing the policy parameters with TRPO. I suspect the authors thus implemented some counter measures in their actual implementation, ensuring numerical stability, that would be nice to know. Furthermore, while the evaluation is extensive, covering a large amount of different domains, the effect of random initialization of the networks seems to be neglected. To that end it would be valuable to see error bars for Figures 1-2 and a note on how stable the presented procedure is with respect to initialization and hyperparameter choices.  Apart from that I only have a few minor comments: - While the paper is well written articles are sometimes missing or wrongly placed, it would be a shame if these small mistakes make it into the paper, here are the ones I found:   Abstract:   - "without interaction with expert or access to reinforcement signal" -> "without interaction with expert or access to a reinforcement signal"   Introduction:   - "Given that learner’s true goal often is" -> "Given that the learner’s true goal often is"   - "a technique from deep learning community that has" -> "a technique from the deep learning community that has"   - "will in fact find that \psi plays a central role in our discussion, not as a nuisance in our analysis." -> "we will in fact find that \psi plays a central role in our discussion, and not only plays the role of a nuisance in our analysis."   Section 5:   - "We are now treat the causal entropy" -> "We now treat the causal entropy"   - "in contrast with linear apprenticeship" -> "in contrast to linear apprenticeship"   - "adaptation of generative adversarial training principle" -> "adaptation of the generative adversarial training principle"   - "addressed the capability of \psi_{GA} of exactly matching occupancy measures" -> "addressed the capability of \psi_{GA} to exactly match occupancy measures"   Section 5.1:   - "denote an empirical expectations over" -> "denote an empirical expectation over"   Section 6:   - "evaluate the imitation performance respect to sample" -> "evaluate the imitation performance with respect to sample" - In Section 6 (and I believe in other places as well) there is a forward reference to the supplementary (as Section 9). It would be good to explicitely write "Section 9 in the supplementary" (and perhaps name the Sections in the supplementary alphabetically) - In Section 5.1 you write that you use the ADAM learning rule but in fact never explicitely mention that you train on mini batches using stochastic gradient descent. Of course using ADAM implies this but it would be better to add it to the sentence.