We present the ADaptive Adversarial Imitation Learning (ADAIL) algorithm for learning adaptive policies that can be transferred between environments of varying dynamics, by imitating a small number of demonstrations collected from a single source domain. This is an important problem in robotic learning because in real world scenarios 1) reward functions are hard to obtain, 2) learned policies from one domain are difficult to deploy in another due to varying source to target domain statistics, 3) collecting expert demonstrations in multiple environments where the dynamics are known and controlled is often infeasible. We address these constraints by building upon recent advances in adversarial imitation learning; we condition our policy on a learned dynamics embedding and we employ a domain-adversarial loss to learn a dynamics-invariant discriminator. The effectiveness of our method is demonstrated on simulated control tasks with varying environment dynamics and the learned adaptive agent outperforms several recent baselines.
Summary:
The submission considers the problem of imitation learning when the dynamics of the expert are not known to the agent and the dynamics of the agent may change frequently. It is however assumed that the agent has access to a parameterized simulator that can simulate the expert dynamics. The parameters for the simulator are not known but are assumed to be drawn from a known distribution.
The proposed method is based on GAIL but uses several modifications:
- A contextual policy is trained that also takes the dynamics-parameters as additional input. At each iteration, a new environment is sampled for performing the policy-rollout.
- A "posterior" prediction network is trained to maximize the likelihood of the parameters that were used for the different roll-outs. This network is used for the test case, where the true dynamics of the agent are not known.
- The discriminator might use features of the state-action input that correlate with the corresponding dynamics. Classifying based on such features may be undesirable because the discriminator might no longer produce useful rewards. In order to address this problem, an additional head is added to the discriminator that outputs a prediction of the dynamic parameters. The prediction error is trained by backpropagation, however by flipping the sign of the gradient at the last shared layer, the features of the discriminator are optimized to be unsuited for predicting the dynamic parameters (the technique is known as Gradient Reversal Layer).
- A VAE-based method for learning latent dynamic parameters is proposed, by training a conditional VAE to reconstruct the next state, where the current state and action are provided as context to the encoder and decoder.
Contribution:
One of the strong-points of the submission is the fact that it features several different, orthogonal contributions. I also think that the considered problem setting is relatively interesting. However, also when considering real applications such as robotics, I am not convinced that explicitly modeling the dynamic changes is necessary. Some existing imitation learning methods focus on a setting where the dynamics of the expert may differ from the agent, but the dynamics of agents do not change. This setting does not require dynamic-contextual policies and seems to be applicable to typical robot applications.
Soundness:
The different components of the proposed methods seem reasonable to me. They do not come with (and arguably do not require) new derivations but seem rather like pragmatic solutions for the encountered problem.
The optimization problem (Eq.2) seems to be formulated slightly inaccurate, because the last term should in my opinion not depend on theta. If I understand correctly, the policy should not maximize the likelihood of the dynamics posterior.
The contrastive regularization loss needs to be better motivated. A high KL in the first term may not necessary be bad, for example, if the confidence in the prediction of (s_0, a_0, s'_0) is lower compared to (s_1, a_1, s'_1). If a similar regularizer has been used in prior work, such work should be referenced. Otherwise, it needs to be motivated.
Presentation/Clarity:
The presentation of the work is arguably the main weakness of the paper.
The submission does not seem polished. It contains a large amount of typos. Figure 2 is confusing and adds little compared to the text description. Also the structure could be improved. For example, the submission introduces the posterior loss and outlines the algorithm before describing the individual components.
The paper uses some techniques such as conditional VAEs [1] or contextual policy search [2]
are used but not described / referenced.
Evaluation:
I like that the different aspects of the proposed method are also evaluated individually. The ablations with respect to the adaptability and GRL are crucial. The evaluation of the performance could be improved. PPO and UP-True use the true reward function, so the only real competitor is a naive GAIL-baseline that uses randomized dynamics during training. I'm not aware of prior work that considers the exact same setting as the manuscript. However, one of the main arguments for inverse reinforcement learning is the claimed generalizability of a reward function as opposed to a policy. I see that learning a new policy after each change in the dynamics may be too costly in some settings. However, comparisons to methods such as AIRL that aim to learn reward functions that are robust to changes in the dynamics would be highly interesting.
[1] Sohn Kihyuk, Honglak Lee, and Xinchen Yan. “Learning Structured Output Representation using Deep Conditional Generative Models.” Advances in Neural Information Processing Systems. 2015.
[2] Marc Peter Deisenroth, Gerhard Neumann, and Jan Peters.  A survey on policy search forrobotics.Foundations and Trends in Robotics, 2(1–2):328–373, 2013.
Comparison: How about IRL, e.g. AIRL?
What about state-only GAIL?
Typos:
Equations are not properly integrated into the sentences (missing punctuations)
"domains, as oppose to one domain."
"Inspired by to GANs"
"and generates a environment"
Figure 5a (legend): "dyanmics"
"that can generalized across"
Algorithmbox: "A environment", "and Generate environment"
"is achieved through 1) allowing"
"the policy is mainly concerned with the end-effector of the latent parameters"
Question: 
What are the network architectures?
According to line 10 of the algorithmbox only the current trajectory is used for updating the dynamics posterior. Why?