Applying reinforcement learning (RL) to physical systems presents notable challenges, given requirements regarding sample efficiency, safety, and physical constraints compared to simulated environments. To enable transfer of policies trained in simulation, randomising simulation parameters leads to more robust policies, but also significantly extends training time. In this paper, we exploit access to privileged information (such as environment states) often available in simulation, in order to improve and accelerate learning over randomised environments. We introduce Attention Privileged Reinforcement Learning (APRiL), which equips the agent with an attention mechanism and makes use of state information in simulation, learning to align attention between state-and image-based policies while additionally sharing generated data. During deployment we can apply the image-based policy to remove the requirement of access to additional information. We experimentally demonstrate accelerated and more robust learning on a number of diverse domains, leading to improved final performance for environments both within and outside the training distribution.
1 .
Summary - Building on top of the domain randomization principle (used to train policies robust to domain-variations) to learn policies which transfer well to new domains, the paper proposes an approach to improve and speed-up learning / training over randomized environments. The paper operates in a settings where the policy to be transferred only has access to observations -- images, etc -- and not the complete underlying state of a (simulated) environment. The underlying idea is to -- (1) maintain two sets of actor-critic networks - a symmetric pair where the actor has access to the underlying state and an asymmetric pair where the actor has access only to image observations; (2) evenly gather experiences from behavioral policies of both actors and store them in a shared replay buffer and (3) learn to align the attention placed by the policies over objects in the environment for the state and observation based actors. The idea is to leverage privileged information about the state (which is strictly more informative compared to observations) to learn robust observation based policies. Experimental results indicate the proposed approach improves generalization performance compared to several ablations of the same on both in-distribution and out-of-distribution environments.
Strengths
- The paper is generally well-written and easy to follow. The authors generally do a good job of motivating the proposed approach by leveraging access to privileged information in the environment / simulation / renderer during training to get more robust observation policies to transfer to novel settings. The proposed approach is presented after appropriately grounding the problem setting and preliminaries and the authors clearly state and evaluate on the axes of research questions they care about.
- The proposed approach is somewhat novel and extends prior work on using shared replay buffers and asymmetric actor critic methods to accelerate training. Although the specific focus is on aligning object-level attention from a state-based actor and an observation based actor, the authors adopt design choices that help in preventing degenerate solutions -- for instance, the object-weighted squared error loss component to learn the observation attention module.
- The experimental results more-or-less support the claims of the paper (however, only in comparison with ablations and 1 baseline) in the sense we see improvements in terms of average returns and sample-efficiency plots. Furthermore, the authors conduct ablations to understand which components of the proposed approach contribute significantly in different environments -- for both extrapolated and interpolated environments.
- Sec 4.6 presents interesting analysis of the learned attention (observation) attention mechanism for both interpolated and extrapolated environments. It seems that attention is generally placed over relevant aspects (objects / links) of the environment and the associated takeaways seem feasible.
Weaknesses
- Having said that, there some weaknesses / questions which if addressed would make the paper stronger and help in increasing the rating of the paper.
- Access to the object-specific attention maps seems to be an assumption that might not scale well across simulators. More realistic / richer simulations (say, 3D reconstructions of indoor rooms) may not always offer this much privileged information -- one might have access to fine-grained reconstructions from multiple viewpoints, but not object level maps. This, combined with the fact that major gains have only been demonstrated over the specified continuous control domains (ignoring the Atari results), makes me slightly concerned about the scalability of the proposed approach in terms of more real-world + applicable domain-transfer scenarios. Maybe a more general approach that learns to match some intermediate representations of the state and observation based actors is a more general approach. Can the authors comment on this?
- Including the entropy loss while training the attention mechanism for the state based actor is justified only through feasible interpretations of the attention visualizations for the JacoReach experiments. I’m curious how important is it for the state-based attention to be sparse? Does it actually affect performance if the entropy loss is not included while learning the state-based attention module?
- Although the paper mentions experience is gathered evenly over the behavioral policies of both the actors it’s slightly unclear how that is being performed without actually referring to the algorithm pseudocode in the appendix. I would encourage the authors to include / move the same to the main paper. It makes the entire pipeline much easier to grasp.
- Transfer to novel environments (not necessarily with domain-shifts) has also been studied in context of providing exploration incentives (see InfoBot - https://arxiv.org/abs/1901.10902) in addition to a (sparse / dense) episodic reward. I would be curious to see how well does APRiL compare to such approaches in a setting where both are applicable - say the MultiRoomNXSY set of experiments in InfoBot (pointed above). This is to understand if the gains obtained from APRiL are very specific to domain-shifts or are largely applicable to any form of novel environment transfer. Can the authors comment on this?
Reasons for rating
Beyond the above points of discussion, I don’t have major weaknesses to point 	out. I generally like the paper. The authors do a good job of identifying the sliver in which they make their contribution and motivate the same appropriately. My major point of concern is centered is around the fact that APRiL (probably) assumes access to much privileged information which may not be generally available across all kinds of environments, etc. My rating of the paper is based on the strengths and weaknesses highlighted above. Addressing / Responding to those appropriately would definitely help in improving the rating of the paper.