Deep Reinforcement Learning has managed to achieve state-of-the-art results in learning control policies directly from raw pixels. However, despite its remarkable success, it fails to generalize, a fundamental component required in a stable Artificial Intelligence system. Using the Atari game Breakout, we demonstrate the difficulty of a trained agent in adjusting to simple modifications in the raw image, ones that a human could adapt to trivially. In transfer learning, the goal is to use the knowledge gained from the source task to make the training of the target task faster and better. We show that using various forms of fine-tuning, a common method for transfer learning, is not effective for adapting to such small visual changes. In fact, it is often easier to re-train the agent from scratch than to fine-tune a trained agent. We suggest that in some cases transfer learning can be improved by adding a dedicated component whose goal is to learn to visually map between the known domain and the new one. Concretely, we use Unaligned Generative Adversarial Networks (GANs) to create a mapping function to translate images in the target task to corresponding images in the source task. These mapping functions allow us to transform between various variations of the Breakout game, as well as between different levels of a Nintendo game, Road Fighter. We show that learning this mapping is substantially more efficient than re-training. A visualization of a trained agent playing Breakout and Road Fighter, with and without the GAN transfer, can be seen in
# Summary
This paper proposes to improve the sample efficiency of transfer learning for Deep RL by mapping a new visual domain (target) onto the training one (source) using GANs. First, a deep RL policy is trained on a source domain (e.g., level 1 of the Atari Road Fighter game). Second, a GAN (e.g. UNIT or CycleGAN) is trained for unsupervised domain adaptation from target images (e.g., level 2 of Road Fighter) to source ones. Third, the policy learned in the source domain is applied directly on the GAN-translated target domain. The experimental evaluation uses two Atari games: i) transfer from Breakout to Breakout with static visual distractors inpainted on the screen, ii) from one Road Fighter level to others. Results suggest that this transfer learning approach requires less images than retraining from scratch in the new domain, including when fine-tuning does not work.
# Strengths
Controlled toy experiments of Deep RL generalization issues:
The experiments on Breakout quantify how badly A3C overfits in this case, as it shows catastrophic performance degradation even with trivial static visual input perturbations (which are not even adversarial attacks). The fine-tuning experiments also quantify well how brittle the initial policy is, motivating further the importance of the problem studied by the paper.
Investigating the impact of different GANs on the end task:
The experiments evaluate two different image translation algorithms: one based on UNIT, the other based on CycleGAN. The results suggest that this choice is key and depends on the target domain. This suggests that the adaptation is in fact task dependent, confirming the direction pursued by others in task-specific unsupervised domain adaptation (cf. below).
# Weaknesses
Discrepancy between quantitative and qualitative results:
The good quantitative results (accumulated rewards) reported in the experiments are not reflected in the qualitative results. As can be seen from the videos, these results seem more to be representative of a bias in the data. For instance, in the Road Fighter videos, one can clearly see that the geometry of the road (width, curves) and dynamic obstacles are almost completely erased in the image translation process. The main reasons the quantitative results are good seem to be i) in the non-translated case the agent crashes immediately, ii) the "translated" image is a wide straight road identical to level 1 where the policy just keeps the car in the middle (thus crashing as soon as there is a turn or a collision with an obstacle). Even in the breakout case, there are catastrophic translation failures for some of the studied variations although the domain gap is static and small. The image translation results look underwhelming compared to state of the art GANs used for much more complex tasks and environments (e.g., the original CycleGAN paper and follow-up works, or the ICLR'18 progressive growing of GANs paper). This might be due to a hyper-parameter tuning issue, but it is unclear why the adaptation results seem not on par with previous results although the paper is in a visually simpler domain (Atari games).
Does not address the RL generalization issues:
Although it is the main goal of the paper, the method is fundamentally side-stepping the problem as it does not improve in any way the policy or the Deep RL algorithm (they are left untouched). It is mapping the target environment to the source one, without consideration for the end task besides tuning GAN hyper-parameters. If the initial policy is very brittle (as convincingly shown in section 2), then just mapping to the source domain does not improve the generalization capabilities of the Deep RL algorithm, or even improves transfer learning: it just enables the policy to be used in other contexts that can be reduced to the training one (which is independent of the learning algorithm, RL or otherwise). So it is unclear whether the main contribution is the one claimed. The contribution seems instead an experimental observation that it might be easier to reduce related domains to the training one instead of retraining a new (specialised and brittle) policy. Existing works have actually gone further, learning jointly the image translation and task network, including for very challenging problems, e.g. in unsupervised sim-to-real visual domain adaptation (e.g., Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks from Bousmalis et al at CVPR'17, which is not cited here).
Experimental protocol:
The experimental conclusions are not clear and lack generality, because the optimal methods (e.g., choice of GAN, number of iterations) vary significantly depending on the task (cf. Table 3 for instance). Furthermore, the best configurations seem selected on the test set for every experiment.
Data efficiency vs actual training efficiency:
The main claim is that it is better to do image translation instead of fine-tuning or full re-training. The basis of that argument is the experimentally observed need for less frames to do the image translation (Table 2). However, it is not clear that training GANs for unsupervised image translation is actually any easier / faster. What about training instability, mode collapse, hyper-parameter tuning, and actual training time comparisons on the same hardware?
# First Recommendation
Using image translation via GANs for unsupervised domain adaptation is a popular idea, used in the context of RL for Atari games here. Although the experiments show that mapping a target visual domain to a source one can enable reusing a deep RL policy as is, the qualitative results suggest this is in fact due to a bias in the data used here and the experimental protocol does not yield general insights. Furthermore, this approach is not specific to RL and its observed generalization issues. It does not improve the learning of the policy or improve its transferability, thus having only limited new insights compared to existing approaches that jointly learn image translation and target task-specific networks in much more challenging conditions.
I believe this submission is at the start of an interesting direction, and requires further work on more challenging tasks, bigger domain gaps, and towards more joint training or actual policy transfer to go beyond this first set of encouraging but preliminary results.
# Post-rebuttal Recommendation
Thanks to the authors for their detailed reply. The clarifications around overfitting, UNIT-GAN in Section 4, and the paper claims are helpful. I  also agree that the quantitative experiments are serious. I have bumped my score by +1 as a result.
Nonetheless, the results still seem preliminary and limited in scope for the aforementioned reasons. The discussion in the comments about the learned policies and transfer are ad-hoc. A lot of the shortcomings mentioned in the review are outright dismissed (e.g., "de facto standard in RL"), downplayed (esp. generalization, which is puzzling for a transfer learning paper), or left for future work.
As there is no strong technical contribution beyond the experimental observations in the current submission, I suggest the authors try to address the GAN shortcomings both mentioned in reviews and their reply, instead of  just observing  / reporting them. As this paper's main focus is to use image translation in the proposed RL setting (with standard GAN and RL methods), I do not think it is just someone else's problem to improve the image translation part. Proposing a technical contribution there would make the paper much stronger and appealing to a broader ICLR audience.  This might also require adding a third game to ensure more generalizable experimental insights.