This paper introduces R2D3, an agent that makes efficient use of demonstrations to solve hard exploration problems in partially observable environments with highly variable initial conditions. We also introduce a suite of eight tasks that combine these three properties, and show that R2D3 can solve several of the tasks where other state of the art methods (both with and without demonstrations) fail to see even a single successful trajectory after tens of billions of steps of exploration.
The paper addresses the problem of exploiting human demonstrations in hard exploration (RL) problems. A new set of challenge tasks is introduced that destroys the performance of very strong baseline systems while highlighting the strength of the new system.
The approach (rarely but consistently training on separately prioritized human experience replays) is well motivated by the shortcomings of past agents (either in overfitting the demonstrated solution or only working in environments with not-too-hard exploration challenges). Where work by others have overspecialized on specific challenge environments (e.g. Montezuma's Revenge with weak stochasticity and observability challenges), this work intentionally dives into difficult territory.
This reviewer moves to accept this top-quality RL paper. The new agent, R2D3, is the primary contribution in combining and outperforming previous SOTA agents. These eight new environments are minor contributions with limited potential for impact on the field, but still make an independently positive contribution.
Questions for authors:
- What would be the present-day approximate retail cost for reproducing the experiments in this paper?
- At the action-rate experienced by the human demonstrators (30fps?), how much wall-clock time represented by 40B actor steps? (40 years?) Is this "making efficient use"?
- Does having highly variable initial conditions really force generalization over environmental configurations or is this wishful thinking / mysticism? To make a direct claim about this, the authors should consider an experimental design where certain classes of initial conditions (e.g. starting on the left side of the map) are withheld during training and evaluated only during testing.
- The finding of small demo ratios as being stronger is exciting, but this result seems to be tied to the specific quantity and quality of demonstrations gathered. Could a more general picture of the role of demonstrations be had by ablating the diversity of representations? The 100 demos in the full case might be degraded to 50, 25, 10, etc while holding the demo ratio fixed. This might effectively vary the weight that demonstrations take in the optimization independently of how often distinct demos are actually seen.
- Can these hard-eight scenarios be parametrically scaled up and down in terms of their exploration effort (possibly by just changing the action granularity / movement speed)? With performance on the new benchmark almost saturated in the first paper based on it, there isn't much room to grow here. In the same way that Montezuma's Revenge was found by scanning the culturally-impactful library of Atari games, perhaps more appropriate and lasting challenges can be found by looking one or more generations forward in the history of commercial console games. Can we play Star Fox? What about SimCity?