Hybrid Reward Architecture for Reinforcement Learning
One of the main challenges in reinforcement learning (RL) is generalisation. In typical deep RL methods this is achieved by approximating the optimal value function with a low-dimensional representation using a deep network. While this approach works well in many domains, in domains where the optimal value function cannot easily be reduced to a low-dimensional representation, learning can be very slow and unstable. This paper contributes towards tackling such challenging domains, by proposing a new method, called Hybrid Reward Architecture (HRA). HRA takes as input a decomposed reward function and learns a separate value function for each component reward function. Because each component typically only depends on a subset of all features, the corresponding value function can be approximated more easily by a low-dimensional representation, enabling more effective learning. We demonstrate HRA on a toy-problem and the Atari game Ms. Pac-Man, where HRA achieves above-human performance.
R5:
Summary:
This paper builds on the basic idea of the Horde architecture: learning many value functions in parallel with off-policy reinforcement learning. This paper shows that learning many value functions in parallel improves the performance on a single main task. The novelty here lies in a particular strategy for generating many different reward functions and how to combine them to generate behavior. The results show large improvements in performance in an illustrative grid world and Miss Pac-man. 
Decision:
This paper is difficult to access. In terms of positives: (1) the paper lends another piece of evidence to support the hypothesis that learning many things is key to learning in complex domains. (2) The paper is well written, highly polished with excellent figures, graphs and results. (3) The key ideas are clearly illustrated in a small domain and then scaled up to Atari. (4) The proposed approach achieves substantial improvement over several variants of one of the latest high-performance deep learning systems.
My main issue with the paper is that it is not well paced between two related ideas from the literature, namely the UNREAL architecture [1], and Diuk’s [2] object oriented approach successfully used many years ago in pitfall—a domain most Deep RL systems perform very poorly on. It’s unclear to me if these two systems should be used as baselines for evaluating hydra. However, it is clear that a discussion should be added of where the Hydra architecture fits between these two philosophical extremes. 
I recommend accept, because the positives outweigh my concerns, and I am interested to here the author’s thoughts.   
The two main ideas to discuss are auxiliary tasks and object-oriented RL. Auxiliary tasks are also built on the Horde idea, where the demons might be predicting or controlling pixels and features. Although pixel control works best, it is perhaps too specific to visual domains, but the feature control idea is quiet general. The nice thing about the UNREAL architecture [1] is that it was perhaps one of the first to show that learning many value functions in parallel can improve performance on some main task. The additional tasks somehow regularize the internal state resulting in a better overall representation. The behavior policy did not require weighting over the different value functions it simply shared the representation. This is a major point of departure from Hydra; what is the best design choice here? Much discussion is needed.
Another natural question is how well does pixel control work in Miss Pac-man, this was not reported in [1].
I see hydra as somewhat less general than auxiliary tasks idea. In Hydra a person must designate in the image what pixel groupings correspond to interesting value functions to learn. One could view Hydra as the outcome of UREAL pixel control combined with some automatic way to determine which pixel control GVFs are most useful for sculpting the representation for good main task behavior. In this way Hydra represents a benchmark for automatic GVF construction and curation.
On the other extreme with have Diuk’s [2] object oriented approach in pitfall. Assuming some way to extract the objects in an Pitfall screen, the objects were converted into a first-order logic representation of the game. Using this approach, optimal behavior could be achieved in one episode. This is interesting because SOTA deep-learning systems don't even report results on this domain. Hydra is somewhat like this but is perhaps more general.
Carefully and thoughtfully relating the Hydra approach to these extremes is critical for accurately discussing the precise contribution of hydra and this paper under review here, and I invite the authors to do that.  
    
The final result of the paper (honestly admitted by the authors) exploits the determinist of Atari domain to achieve super-human play in miss packman. This result shows that A3C and others do not exploit this determinism. I am not sure how generally applicable this result is. 
Another method not discussed, Model Free Episodic Control [3], performs well in Miss Pac-man—much better than A3C. Episodic control defines a kernel function and memory built from agent states generated by a deep network. This allows the system to quickly the query the values of related previously seen states, resulting in much faster initial learning in many domains.  I would be interested to see the performance of episodic control in miss pac-man with both representations.
It is interesting that DQN does not seem to exploit the additional domain knowledge in the fruit experiment. I am trying to figure out the main point here. Is the suggestion that we will usually have access to such information, and hydra will often exploit it more effectively? Or that prior information will usually be of this particular form that hydra uses so effectively?  Could unreal’s aux tasks make use of manually ignoring irrelevant features? If that is not feasible in UNREAL, this would be yet another major difference between the architectures! This raises an interesting question: how often can this feature irrelevance be automatically observed and exploited without causing major problems in other domains. It seems like UNREAL’s independent GVFs could eventually learn to ignore parts of it’s shared representation. 
Low-dim representations: this is a meme that runs throughout the paper, and a claim I am not sure I agree with. Several recent papers have suggested that the capacity of deep learning systems is massive and that they can effetely memorize large data-sets. Do you have evidence to support this seemingly opposite intuition, outside of the fact that Hydra works well? This seems like a post-hoc explanation for why hydra works well. This should be discussed more in the paper
[1] Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., & Kavukcuoglu, K. (2016). Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397.
[2] Diuk, C., Cohen, A., & Littman, M. L. (2008, July). An object-oriented representation for efficient reinforcement learning. In Proceedings of the 25th international conference on Machine learning (pp. 240-247). ACM.
[3] Blundell, C., Uria, B., Pritzel, A., Li, Y., Ruderman, A., Leibo, J. Z., ... & Hassabis, D. (2016). Model-free episodic control. arXiv preprint arXiv:1606.04460.
   
Small things that did not effect the final decision:
line 23: performed > > achieved 
26: DQN carries out a strong generalisation > > DQN utilizes global generalization—like any neural net—
28: model of the optimal value function. I don’t think you mean that
230: how important are the details of these choices? Could it be -500 or -2000?