We present Decentralized Distributed Proximal Policy Optimization (DD-PPO), a method for distributed reinforcement learning in resource-intensive simulated environments. DD-PPO is distributed (uses multiple machines), decentralized (lacks a centralized server), and synchronous (no computation is ever 'stale'), making it conceptually simple and easy to implement. In our experiments on training virtual robots to navigate in Habitat-Sim (Savva et al., 2019), DD-PPO exhibits near-linear scaling -achieving a speedup of 107x on 128 GPUs over a serial implementation. We leverage this scaling to train an agent for 2.5 Billion steps of experience (the equivalent of 80 years of human experience) -over 6 months of GPU-time training in under 3 days of wall-clock time with 64 GPUs. This massive-scale training not only sets the state of art on Habitat Autonomous Navigation Challenge 2019, but essentially 'solves' the task -near-perfect autonomous navigation in an unseen environment without access to a map, directly from an RGB-D camera and a GPS+Compass sensor. Fortuitously, error vs computation exhibits a power-law-like distribution; thus, 90% of peak performance is obtained relatively early (at 100 million steps) and relatively cheaply (under 1 day with 8 GPUs). Finally, we show that the scene understanding and navigation policies learned can be transferred to other navigation tasks -the analog of 'ImageNet pre-training + task-specific fine-tuning' for embodied AI. Our model outperforms ImageNet pre-trained CNNs on these transfer tasks and can serve as a universal resource (all models + code will be publicly available).
The contribution of this paper is twofold. First the authors propose and implement and new scalable RL training procedure they call 'Decentralized Distributed Proximal Policy Optimization' (DD-PPO). This method extends PPO in a way that is 'synchronous' and 'distributed': no parameter server exists to asynchronously collect experience from worker agents and instead an 'explicit communication state' exists during which all workers communicate gradient updates between one another to update parameters based on their experiences. However, naively implementing this approach is limited by 'stragglers', the slowest of the workers that all other agents need to wait for. To overcome this limitation, the authors add a 'preemption threshold' which halts worker rollouts after a certain percentage has completed. The authors use this procedure to tackle the task of PointGoal navigation, in which an agent tries to reach a point in space specified by its relative location to the agent. They achieve state-of-the-art performance (arguably 'mastery') on this task. Finally, the authors show that their learned policy transfers reasonably well to two other navigation tasks, 'flee' and 'exploration'. Code is also provided.
This is an impressive paper. The algorithmic contribution is useful and the results are state-of-the-art. The paper is written very clearly, and it was a pleasure to read. The investigation of the task of PointGoal Navigation, and the impact of various changes on the overall performance of the system . The transfer tasks, to the 'flee' and 'exploration' tasks, were interesting, and further reinforced the results in the paper.
My only high-level 'complaint' is that the title might be tempered: 'Mastering PointGoal Navigation' is a very strong claim. The results in the paper are very impressive, but 'Mastering', in my mind, implies (among other things) that the agent is guaranteed to reach its goal, though I understand that it is perhaps a minor point, considering the failure rate is systematically <3%. I also make a comment later about how the Matterport 3D environments may be a biased set, and that there are almost certainly environments that *would* make this system fail, for one reason or another. I am willing to accept the title as-is (and could be overwritten by other reviewers), but I would like some discussion on a more appropriate tile, if possible.
I recommend this paper for acceptance.
Minor comments:
- Introduction: "Thus, there is a need to develop a different distributed architecture." It is unclear how this sentence logically follows from the previous sentence. I am on board with the vision in general, and it is quite clear to me: having a single parameter server is limiting. However, that the move from CPU- to GPU-based agents is what reveals this limitation is not quite so clear. Reworking this paragraph (or perhaps adding a sentence) to make this clearer would be helpful.
- Introduction: "This means there is no scope for mistakes of any kind --- no wrong turn at a crossroad, ..." This is a fascinating point, though I feel reflects a bias in the types of environments that appear in the training data set. It is a defining behavior of intelligent embodied agents that they are capable of recovering from their mistakes; in general, real-world environments and navigation tasks have inherent ambiguity that cannot be resolved without exploration (a 'maze' is an extreme example of this). One question comes to mind: if the environments were ambiguous in this way, *could* the agent recover from its mistakes. There is not much evidence in the paper one way or the other. Since there are few examples in the data that seem to bring about such scenarios, it is not obvious how well this agent would perform in the face of this uncertainty. However, the results later in the paper about how "No GPS+Compass remains unsolved" is related to this point. A comment (or a figure in the Appendix) from the authors describing some of the failure cases would be instructive: do these failures occur in environments with such an ambiguity or, conversely, are there some examples in which the agent succeeds in reaching the goal but must overcome failure during its travel?
- The acronym 'SL' (for Supervised Learning) is not defined and is used only once. Prefer using 'supervised learning'.