CDIL_ICLR_2020 (10).pdf
We study the question of how to imitate tasks across domains with discrepancies such as embodiment and viewpoint mismatch. Many prior works require paired, aligned demonstrations and an additional RL procedure for the task. However, paired, aligned demonstrations are seldom obtainable and RL procedures are expensive. In this work, we formalize the Cross Domain Imitation Learning (CDIL) problem, which encompasses imitation learning in the presence of viewpoint and embodiment mismatch. Informally, CDIL is the process of learning how to perform a task optimally, given demonstrations of the task in a distinct domain. We propose a two step approach to CDIL: alignment followed by adaptation. In the alignment step we execute a novel unsupervised MDP alignment algorithm, Generative Adversarial MDP Alignment (GAMA), to learn state and action correspondences from unpaired, unaligned demonstrations. In the adaptation step we leverage the correspondences to zero-shot imitate tasks across domains. To describe when CDIL is feasible via alignment and adaptation, we introduce a theory of MDP alignability. We experimentally evaluate GAMA against baselines in both embodiment and viewpoint mismatch scenarios where aligned demonstrations don't exist and show the effectiveness of our approach.
Summary
This paper proposes a method to perform alignment between demonstrations in different domains in order to adapt a policy from expert domain to the test domain. They also include a formalism that studies alignments of different MDPs.
Strengths
1) The paper presents a formal framework to study many recent methods in cross-domain/cross-view imitation learning with a common lens. 
2) The solution proposed by the authors might have a big practical advantage for cross-domain imitation learning settings. They claim that they do not need to learn the policy using RL in the new domain as they learn the action mapping between the two domains to retrieve the corresponding action in new domain based on the action in expert domain. This is a very interesting result.
3) The solution is quite modular with 3 important parts: i) state mapping network, ii) action mapping network, iii) dynamics model. 
Weakness
The biggest weakness of the paper is in the evaluation section. 
1) Issues with baselines:
i) There seems to be some problem with UMA. In Figure 4, it cannot figure out the alignment with self domain. Why is that the case? Where is the cartpole in UMA row for pen-cartpole alignment ? Given such poor performance during alignment is it fair to compare to UMA in the subsequent section?
ii) Why are the other methods not able to align well when there is no domain shift? 
iii) How is dynamic time warping actually used to calculate the state correspondences for training IfO and IF? Is it calculated on top of the states?
iv) How are the policies trained for the transferability task? Which RL algorithm? "Baselines fail to learn the writing task as an inaccurate proxy reward function harms performance." 
2 aspects are involved here:
1) initializing from alignment network
2) using rewards from the state representation to train the policy.
Do the algorithms still fail if the true reward is used with the alignment initialization? 
Reward scaling is known to be important for RL algorithms[1]. May be features from these algorithms have to be scaled to scale the rewards for the RL algorithms to work.
2) Issues with environments:
i) The environments considered for domain transfer seem quite trivial for the alignment between them to be a big problem. For example, the authors present the task R2W: reacher2-tp that has a "third person" state space with a 180 camera angle offset. I am not sure how this rotation changes the state for the policy that uses states as input. One of the states of reacher is vector between goal and state which should still be informative enough to solve the task. Presumably the rotation creates a different view for the image based experiments. There also the transformation is quite easy for the spatial autoencoder given that it has access to coordinates through the spatial softmax layer. 
ii) It is unclear what the R2W task entails or how difficult it is. 
"The transfer task is writing letters as fast as possible. The transfer task differs from the alignment tasks in two key aspects: the end effector must draw a straight line from a letter’s vertex to vertex and minimally slow down at the vertices. "
It seems there is a term that penalizes the policy from doing things quickly. Some questions regarding this task:
1) How is this task specified? 
2) Is agent rewarded for slowing down too?  Does the goal of the reacher and the state representation corresponding to the vector between end effector and goal get updated once it reached a particular point? Does the agent have to write one particular letter? Why do the other baselines not work at all on this task?  
iii) Is it possible to run experiments on environments presented in Invariant Features paper so that the importance of CDIL can be assessed better.
4) "HIGH-LEVEL COMPARISON TO BASELINES" section is misleading. While the different baselines (TCN, IfO, TPIL) were developed in the context of view mismatch that does not mean the algorithm cannot be applied for embodiment mismatch. For IF also the same idea can be applied for viewpoint mismatch. TCN also proposed single-view version which can be used to learn representations without paired data. Hence, it is not right to say TCN can't work with unpaired alignment data. It also needs to be stated in this section that CDIL requires access to actions while some baselines like TCN do not need actions that can be used for training.  
Questions
1) How were the alignment videos generated?
Decision
This paper has the potential to be an important paper in this field. But at this point needs further empirical evaluation with stronger baselines and known benchmark environments.
Minor Comments:
1)  "boltmzman machine reconstruction error" - Boltzmann
 
References
[1] "Deep Reinforcement Learning That Matters." Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger