InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations
The goal of imitation learning is to mimic expert behavior without access to an explicit reward signal. Expert demonstrations provided by humans, however, often show significant variability due to latent factors that are typically not explicitly modeled. In this paper, we propose a new algorithm that can infer the latent structure of expert demonstrations in an unsupervised way. Our method, built on top of Generative Adversarial Imitation Learning, can not only imitate complex behaviors, but also learn interpretable and meaningful representations of complex behavioral data, including visual demonstrations. In the driving domain, we show that a model learned from human demonstrations is able to both accurately reproduce a variety of behaviors and accurately anticipate human actions using raw visual inputs. Compared with various baselines, our method can better capture the latent structure underlying expert demonstrations, often recovering semantically meaningful factors of variation in the data.
Paper Summary:
This paper focuses on using GANs for imitation learning using trajectories from
an expert. The authors extend the GAIL (Generative Adversarial Imitation
Learning) framework by including a term in the objective function to incorporate latent structure (similar to InfoGAN). The authors then proceed to show that using their framework,
which they call InfoGAIL, they are able to learn interpretable latent structure
when the expert policy has multiple modes and that in some setting this robustness allows them to outperform current methods.
Paper Overview:
The paper is generally well written. I appreciated that the authors first demon-
started how the mechanism works on a toy 2D plane example before moving
onto more complex driving simulation environment. This helped illustrate the
core concepts of allowing the learned policy to be conditioned on a latent variable
in a minimalistic setting before moving on to a more complex 3D driving simulation.
My main concern with the paper is that, in integrating many different ideas, 
it was hard to distil the core contribution of the paper which, as I understood, 
was the inclusion of latent variables to learn modes within an overall policy when using GAIL.
Major Comments:
1. In integrating many different ideas, it was difficult to distil the core 
contribution of this paper (the inclusion of latent variables to learn modes 
within a policy). As an illustration of this problem, In subsection 3.1 the authors refer 
to "InfoGAIL" in its simplest form, but the supplementary material they present a more 
complex algorithm which includes weight clipping, Wasserstein GAN objective as well as an
optional term for reward augmentation and this is also called "InfoGAIL". I felt the paper 
could beneft from concretely defining InfoGAIL in its most simple form, and then when 
adding additional enhancements later and making it clear this is InfoGail with these extra enhancements.
2. Linked to my first point, I felt the paper would benefit from an algorithm
box for the simplest form of InfoGAIL in the paper itself (not supplement-
tary). This would add to the readability of the paper and
allow a reader not too familiar with all the details of GANs to quickly
grasp the main ideas of the algorithm. I'd suggest adding this before the
section 4 on optimization. Then keeping the InfoGAIL with extensions in
supplementary material (and renaming it accordingly there).
3. The authors show how InfoGAIL works in an idealized setting where the number of modes are known. 
It would be useful to see how InfoGAIL behaves when the number of modes is not known and the number of latent variables 
are set incorrectly. For example, in the 2D plane environment there are 3 modes. What happens if we use InfoGail 
but set the number of latent variables to 2, 4, 5, 10 etc. Can it still infer the modes of the expert policy?
Is the method robust enough to handle these cases?
Minor Comments:
1. subsection 4.1 refers to reward augmentation. But the experiment in sub-
section 5.2 don't mention if reward augmentation is used to obtain results
(reward augmentation is only mentioned for experiments in subsection
5.3). It's not clear if the authors use reward augmentation in this experiment 
and if so how it is defined.
2. The heading of the paper is "Inferring The Latent Structure of Human
Decision-Making from Raw Visual Inputs". Does InfoGAIL only work
in domains where we have raw visual inputs. Also we may assume the method would work
if the expert trajectories do not come from a human but some other noisy source.
The authors may want to revise the title to make the generality of their framework more
clear.