How do humans acquire a meaningful understanding of the world with little to no supervision or semantic labels provided by the environment? Here we investigate embodiment and a closed loop between action and perception as one key component in this process. We take a close look at the representations learned by a deep reinforcement learning agent that is trained with visual and vector observations collected in a 3D environment with sparse rewards. We show that this agent learns semantically meaningful and stable representations of its environment without receiving any semantic labels. Our results show that the agent learns to represent the action relevant information extracted from pixel input in a wide variety of sparse activation patterns. The quality of the representations learned shows the strength of embodied learning and its advantages over fully supervised approaches with regards to robustness and generalizability.
Paper summary:
This is an empirical study of the representations learned by a reinforcement learning agent. An agent is trained, using a standard RL algorithm, to solve puzzles by navigating through a 3D visual environment (Unity obstacle tower challenge). The analyses in the paper show that the visual representations of the trained agents are sparse and cluster according to the actions performed by the agent. The goal of the paper is to show that these features are due to the embodied nature of the agent. Specifically, the paper states that “the quality of the representations learned shows the strength of embodied learning and its advantages over fully supervised approaches with regards to robustness and generalizability”.
Decision:
I suggest to reject this paper. While the topic is interesting and the paper is clearly written, there is a lack of control/comparison experiments, such that the paper’s conclusions are not backed up by the analyses. However, with more experiments, I think this line of research has large potential.
Further justification for the decision:
My main criticism is that the paper claims to show that embodiment is important for representation learning, but never actually compares representations learned by an embodied agent to representations learned in some other way.
Comparing to a random network is a sanity check, but not sufficient to support the paper’s claims.
One way to experimentally dissociate embodiment/agency from supervised learning would be to train two separate models, one that is “embodied”/active, and another that gets the same sensory input, but without embodiment (“passive”). The passive network could be trained using the sensory inputs recorded while training the active network. The passive network could be trained to predict the value and/or the actions of the active network. Thus, the passive network would be trained in a supervised way, whereas the active network would be trained by RL (i.e. in an embodied way).
The representations of the two networks could then be compared using the analyses used in the paper. This setup would experimentally isolate the effect of embodiment. Without such comparisons, it is unclear whether representations learned in a supervised way would be any different from those learned by RL.
In addition to the descriptive analyses presented in the paper, a transfer learning approach could test whether there is actually a functional difference between the “embodied” and the supervised representations: Take the representations of the “active agent” and the “passive agent” and freeze the weights. Then re-initialize and re-train only the dense layer before the action probabilities on the RL task, leaving everything else frozen. Does the model from the “active agent” do better on the RL task? This would suggest that the embodied agent learned better representations.
Minor comment: The website contains a link to a YouTube profile that is not completely anonymous (it contains the first name and a profile photo). While I did not identify the authors when visiting the paper website, I recommend removing links to personal YouTube profiles and create submission-specific anonymous accounts.