On Learning Intrinsic Rewards for Policy Gradient Methods
In many sequential decision making tasks, it is challenging to design reward functions that help an RL agent efficiently learn behavior that is considered good by the agent designer. A number of different formulations of the reward-design problem have been proposed in the literature. In this paper we build on the Optimal Rewards Framework of  that defines the optimal intrinsic reward function as one that when used by an RL agent achieves behavior that optimizes the task-specifying or extrinsic reward function. Previous work in this framework has shown how good intrinsic reward functions can be learned for lookahead search based planning agents. Whether it is possible to learn intrinsic reward functions for learning agents remains an open problem. In this paper we derive a novel algorithm for learning intrinsic rewards for policy-gradient based learning agents. We compare the performance of an augmented agent that uses our algorithm to provide additive intrinsic rewards to an A2C-based policy learner (for Atari games) and a PPO-based policy learner (for Mujoco domains) with a baseline agent that uses the same policy learners but with only extrinsic rewards. We also compare our method with using a constant "live bonus" and with using a count-based exploration bonus (i.e., pixel-SimHash). Our results show improved performance on most but not all of the domains.
This work attempts to use learnable intrinsic rewards in addition to conventional extrinsic reward from the environment to boost the agent performance calculated as conventional returns as the cumulative extrinsic rewards. This work can be seen as a variant of previous reward shaping and auxiliary rewards works but cannot be a more general version because though with more general mathematical form, it loses the consideration of domain knowledges, the key insights of previous works. Compared with its closest related works [Sorg et al. 2010, Guo et al. 2016], the method proposed here can be used in bootstrapping learning agents rather than only planning agents (i.e., Monte Carlo sampling of returns).         Strengths:  1. This work implements an intuitively interesting idea that automatically shaping the rewards to boost learning performance. And compared with closely related previous works, it is more general to be used in modern, well performed learning methods (e.g., A2C and PPO). 2. The presentation is clear and easy to follow.  3. The experiment evaluation shows obvious improvement in the performance of the proposed method.         Weakness:   1. The technical design is in lack of theoretical support. Even if we accept the design that the policy parameters are updated to optimize the combinational returns while the intrinsic reward parameters are updated to optimize only the extrinsic returns, which itself needs more justification (though has been taken in Sorg et al. 2010), the update derivation for the intrinsic reward parameters (i.e., Eq.7-10) is hardly convincing.   2. Though the superior performance is shown in the experiment, there are still worse performance cases (e.g., the forth panel in Fig.4), which lacks explanation.  3. As pointed out in Sec.1, the basic assumption to use reward shaping (or adding intrinsic reward) is to "change for better or for worse the sample (and computational) complexity of the RL agent learning from experience in its environment using the transformed reward function". But throughout the work, no theoretical, or experimental analysis can support this assumption based on the proposed design. One possible way to repair it will be to use simple thought/simulation experiments to showcase this, which will be much clearer than the performance shown in complicated benchmarks.