Recent advances in deep reinforcement learning require a large amount of training data and generally result in representations that are often over specialized to the target task. In this work, we study the underlying potential causes for this specialization by measuring the similarity between representations trained on related, but distinct tasks. We use the recently proposed projection weighted Canonical Correlation Analysis (PWCCA)to examine the task dependence of visual representations learned across different embodied navigation tasks. Surprisingly, we find that slight differences in task have no measurable effect on the visual representation for both SqueezeNet and ResNet architectures. We then empirically demonstrate that visual representations learned on one task can be effectively transferred to a different task. Interestingly, we show that if the tasks constrain the agent to spatially disjoint parts of the environment, differences in representation emerge for SqueezeNet models but less-so for ResNets, suggesting that ResNets feature inductive biases which encourage more task-agnostic representations, even in the context of spatially separated tasks. We generalize our analysis to examine permutations of an environment and find, surprisingly, permutations of an environment also do not influence the visual representation. Our analysis provides insight on the overfitting of representations in RL and provides suggestions of how to design tasks that induce task-agnostic representations.
This article studies the similarities between the learned representations for different tasks when trained using reinforcement learning algorithms. The ultimate question that this study tries to answer is an interesting one. Namely, how much can representations learned by training on one task be beneficial for learning other tasks? A high interdependence between the representations can lead to a more successful transfer of knowledge between tasks.
The authors are further interested in studying the properties that influence this relationship, which depends on the elements of training as well as attributes of the tasks themselves.
However, I believe that the results are not strong enough to support the claims that are stated in the paper and the limited scope of the environments tested does not make a convincing case that the results will be generalizable much beyond these scenarios. Therefore, in the current state, I think this paper should be rejected.
Firstly, the paper states that:
> ".. if the distance between models trained on different tasks is the same as that between models trained on the same task, representations are task-agnostic."
This seems to be a key argument in the paper. However, I believe that this argument is based on the assumption that representations learned for a single task are indeed highly similar. I think this assumption requires some sort of support as reinforcement learning algorithms are known to be highly inconsistent even in reaching similar solutions. Therefore, one cannot take for granted that the learned representations would be similar in any way.
Second, the authors claim in Section 5 that the random splits have little impact on the learned representation, but in Section 6 claim that the spatially disjoint splits have a noticeable impact on the representations. Without any measure of the impact, looking at figures 1b and 3a, I'm not convinced that this distinction is so obvious. The situation is worse when looking at the figures in the appendix, namely figures A3 and A5. If someone were to swap these two figures, my untrained eye would not be able to tell the difference.
I suggest that the authors spend more time explaining their reasoning as to why these results are significant enough to support the claims. Also, the text should be improved if it is to be accepted. There exist many problems ranging from small typos (simlate -> simulate, reuse-ability -> reusability, and "?." -> "?") to sentences that need to be reworked. Some figure legends/captions can also be improved to include more information, such as explaining what exactly the shaded regions represent (I'm guessing one standard deviation from the mean over some unknown replicates), or in Fig 3b making clear whether "A -> B" is done with "New Policy" or "Fine-tuned Policy". I am also curious as to why only one of these scenarios was experimented with in sections 6 and 7.