Domain adaptation is an open problem in deep reinforcement learning (RL). Often, agents are asked to perform in environments where data is difficult to obtain. In such settings, agents are trained in similar environments, such as simulators, and are then transferred to the original environment. The gap between visual observations of the source and target environments often causes the agent to fail in the target environment. We present a new RL agent, SADALA (Soft Attention DisentAngled representation Learning Agent). SADALA first learns a compressed state representation. It then jointly learns to ignore distracting features and solve the task presented. SADALA's separation of important and unimportant visual features leads to robust domain transfer. SADALA outperforms both prior disentangled-representation based RL and domain randomization approaches across RL environments (Visual Cartpole and DeepMind Lab).
Pros:
This paper proposed a new method for zero-shot transfer learning under the reinforcement learning setting. The use of attention weights to regularize the latent states was fairly interesting.
Cons:
Limited applicability of the proposed methods
- The paper was restricted in a setting where rewards, actions, and true states were identical between source and target environments, and only the observed states differed due to differing renderers. Working under such a restricted setting was interesting in its own right, but it might also lead to limited applicability of the proposed method in the real-world setting.
- The proposed method focused on solving a very specific problem: learning a dis-entangled latent representation for images. As a result, the potential impact of the proposed methods could be minimal.
Limited technical novelty
- The proposed method, SADALA, was built on top of Higgins et al., 2017 (DARLA). The only difference was an added attention layer to the learning of latent states. As a result, the novelty of the proposed method was very incremental and limited from a technology perspective.
- Even with additional attention layer, the paper could have performed a more thorough study to help the readers understand and appreciate the idea. For example, this paper didn’t discuss the tradeoff between training SADALA over separate stages, versus training it from end to end. For example, why the weights of the pre-trained beta-VAE had to be frozen and used as weights in the state representation stage.
Insufficient experiments
-More thorough discussion of the qualitative results should be helpful to understand whether the attention weights helped the model to focus on the right thing. For example, this paper did study the quality of reconstruction in Figure 3-5 of the proposed method. When comparing Figure 3 and Figure 5, it appeared to me that the reconstructed the angle of the pole was different from the original one. And it seemed like attention weights did successfully ignored the color of the cart and pole, but it ignored the angle of the pole, which should be important to the learning task. Unfortunately, the paper didn't further explain the implication of such misrepresentation.
-Quantitative results 
* It would be interesting to all compare the proposed methods against model-agonistic methods like MAML
* It would be useful to include confidence intervals over different tasks.
* It would be useful to compare different methods with different parameter settings
* The authors mentioned “Visual Pendulum tasks” but didn’t include them in the paper
Reproducibility
- It's unclear to me how reproducible the research conducted in this paper was, and it would be useful to open source the code used to conduct the experiments.