We show that a critical problem in adversarial imitation from high-dimensional sensory data is the tendency of discriminator networks to distinguish agent and expert behaviour using task-irrelevant features beyond the control of the agent. We analyze this problem in detail and propose a solution as well as several baselines that outperform standard Generative Adversarial Imitation Learning (GAIL). Our proposed solution, Task-Relevant Adversarial Imitation Learning (TRAIL), uses a constrained optimization objective to overcome task-irrelevant features. Comprehensive experiments show that TRAIL can solve challenging manipulation tasks from pixels by imitating human operators, where other agents such as behaviour cloning (BC), standard GAIL, improved GAIL variants including our newly proposed baselines, and Deterministic Policy Gradients from Demonstrations (DPGfD) fail to find solutions, even when the other agents have access to task reward.
The paper addresses the acute problem of irrelevant discrimination in adversarial imitation learning methods. Ideally, the discriminator in such methods should extract task-dependent features and base its discrimination rule upon them only. However, in practice, the opposite usually occurs. For example, if the state distributions of the expert and agent come from different modalities, e.g., different background light, different POV camera, etc., then almost surely the discriminator will base its classification rule on such differences. In turn, the reward signal will be (constant and) irrelevant. This is a well-known problem, that indeed received little attention in previous work. 
As a solution, the authors propose to maintain an invariance set representing task-irrelevant states. For example, states of a random policy or states from the beginning of an episode. The authors suggest using the invariance set as a regularization mechanism to the discriminator by punishing it for correctly classifying examples from the invariance set. The idea behind this regularization is to distinguish between expert examples that actually represent the task and expert examples that do not. This is achieved by punishing for the discriminator's accuracy over the invariance set.
In addition, the authors present two additional improvements: data augmentation and early actor stopping.
The paper.
1. I very much like the problem the authors choose to address. I believe that this is a dominant problem that received little attention so far. The reason is that so far, GAIL style methods used experts and agents that come from the same domains (same simulation), thus "hiding" such problems. As experiments scale up to real-world setups, this is no longer the case and such problems arise.
2. Overall, I find the experiments section very confusing. The authors propose a formal improvement (the invariance set) which they apply to their agent, as well as two informal improvements (actor early stopping and data augmentation) which they apply selectively for their agent and the baseline methods. As a result, there exist a large number of variations, according to what improvements are applied to what method. Overall, I find it hard to follow the results and understand the contribution of the different improvements.
3. It is not clear if TRAIL and GAIL are using the same architecture and algorithm. Apparently, the authors could have used GAIL's original policy gradient algorithm with their invariance set modification. If an improvement was evident in such a comparison, then it would have been easier for me to understand the significance of the proposed method. Currently, there are too many differences between the baseline and TRAIL to clearly say that the invariance set made the difference.
4. It is not clear why the constraint in Eq. 2 is placed over the sum of the two terms and not on each term separately since each term is defined over a different state distribution. What would you expect to see different if the constraint was placed on the two terms separately?
5. In the same matter, why do you need the 0.5 factor?
6. The invariance set is referred to extensively before it is first defined. Therefore, the reader is forced to toggle back and forth multiple times in section 4 before understanding equations 2 and 3. Consider giving at least an intuition about the invariance set beforehand to make the reading more fluent.
7. The conclusions section is minimal to non-existing while there are many conclusions to draw from the experiments.
8. Data augmentation was previously studied in the context of adversarial imitation. See for example "Visual imitation with reinforcement learning using recurrent Siamese networks" by Berseth et.al.
9. Overall assessment: among the three modifications the authors propose, I would be glad to see if the invariance set modification alone can make a difference. The other two modifications are either not novel (see Berseth et.al.), or highly heuristic ("...we restart an episode if the discriminator score at the current step exceeds the median score of the episode so far for T_patience consecutive steps..."). However, it seems that the informal improvements (augmentation and early stopping) play a crucial role in the success of TRAIL (... It drastically improves the baseline GAIL agent, and is necessary to solve any of the harder tasks..."). It is absolutely fine that the authors stress the importance of the informal improvements but it seems that they are dominating the contribution of the main ingredient.
I also find it problematic that the construction of the invariance set is not straightforward ("...the selection of the invariance set is a design choice..."). When added to the way that the data augmentation is carried (sensor dependent), and to the fact the the early stopping mechanism seems dubious for some tasks (I can think of several tasks where early stopping will not work), this method is far from being a "plug-and-play" solution. I.e., the paper offers general guidelines on how to solve imitation tasks but not a concise algorithm.
In order for me to vote for acceptance, I need the paper to be more concise. In its current state, I see it more as general guidelines for training adversarial imitation algorithms.
However, as said in the beginning, I very much believe in the problem that the authors present and encourage them to provide an experiment where the only modification between the baseline and the proposed method is the usage of an invariance set. Success in such an experiment will clearly change my mind.
There are also some presentation issues, e.g cluttered graphs, that can be addressed.