Reward Mapping for Transfer in Long-Lived Agents
We consider how to transfer knowledge from previous tasks (MDPs) to a current task in long-lived and bounded agents that must solve a sequence of tasks over a finite lifetime. A novel aspect of our transfer approach is that we reuse reward functions. While this may seem counterintuitive, we build on the insight of recent work on the optimal rewards problem that guiding an agent's behavior with reward functions other than the task-specifying reward function can help overcome computational bounds of the agent. Specifically, we use good guidance reward functions learned on previous tasks in the sequence to incrementally train a reward mapping function that maps task-specifying reward functions into good initial guidance reward functions for subsequent tasks. We demonstrate that our approach can substantially improve the agent's performance relative to other approaches, including an approach that transfers policies.
This paper presents a model for transfer where 
      synthetic reward functions are learned from previous tasks, and are mapped 
      to new tasks to produce synthetic reward functions which speed up 
      learning. This application of the relatively recent idea of a synthetic 
      reward function that improves the performance of an underlying "utility" 
      reward function to the transfer scenario is new and interesting, and the 
      experiments are well thought out, thorough and reasonably convincing. 
      The major criticism I had of the paper is that it does not compare 
      to other methods of transfer. The authors are incorrect that policy 
      transfer is not possible in their setting; that is why sub-policies, in 
      the form of options, are often transferred instead of entire policies. 
      (Speaking of which, the authors should probably cite Lisa Torrey's work on 
      policy transfer.) In this case though it's OK to skip that comparison 
      because you could use either, both, or neither types of transfer in 
      combination, so the comparison adds information but is not critical. 
      However, the failure to compare against a similar reward shaping 
      scheme is more problematic. These two methods are effectively solving the 
      same problem, except that reward shaping does not change the ultimate 
      solution and a synthetic reward function does. In my opinion, these two 
      methods (even outside of the transfer scenario) have not been adequately 
      compared, which is odd because its trivial to learn a shaping function 
      (it's just a value function initialization, so you're just learning a 
      value function). This lack of comparison leaves me with significant doubts 
      about the whole synthetic reward function enterprise generally. So I think 
      a comparison here - where the shaping function is mapped in the same way 
      that the reward function is - would significantly improve the paper. But 
      perhaps that is too much to ask for in a single paper, especially since 
      the mapped shaping function could be considered a new (though somewhat 
      obvious) method. The paper is very well written and was generally 
      a pleasure to read, though this was spoiled somewhat by the repeated use 
      of parenthetical citations as nouns. The references were poorly formatted 
      in some cases (capitalization on "mdps", etc.)