Lifelong Inverse Reinforcement Learning
Methods for learning from demonstration (LfD) have shown success in acquiring behavior policies by imitating a user. However, even for a single task, LfD may require numerous demonstrations. For versatile agents that must learn many tasks via demonstration, this process would substantially burden the user if each task were learned in isolation. To address this challenge, we introduce the novel problem of lifelong learning from demonstration, which allows the agent to continually build upon knowledge learned from previously demonstrated tasks to accelerate the learning of new tasks, reducing the amount of demonstrations required. As one solution to this problem, we propose the first lifelong learning approach to inverse reinforcement learning, which learns consecutive tasks via demonstration, continually transferring knowledge between tasks to improve performance.
Summary:  This paper considers the problem of lifelong inverse reinforcement learning, where the goal is to learn a set of reward functions (from demonstrations) that can be applied to a series of tasks. The authors propose to do this by learning and continuously updating a shared latent space of reward components, which are combined with task specific coefficients to reconstruct the reward for a particular task.   The derivation of the algorithm basically mirrors the Efficient Lifelong Learning Algorithm (ELLA) (citation [33]). Although ELLA was formulated for supervised learning, variants such as PG-ELLA (not cited in this paper, by Ammar et al. “Online Multi-task Learning for Policy Gradient Methods”) have applied the same derivation procedure to extend the original ELLA algorithm to the reinforcement learning setting. This paper is another extension of ELLA, to the inverse reinforcement learning setting, where instead of sharing policies via a latent space, they are sharing reward functions.    Other Comments:  How would this approach compare to a method that used imitation learning to reconstruct policies, and then used PG-ELLA for lifelong learning?  To clarify, ELIRL can work in different state/action spaces, and therefore also different transition models. However, there is an assumption that the transition model is known. Specifically it would be necessary to have the transition model to solve for pi^{alpha^(t)} when computing the hessian in Equation (5).   I liked the evaluation procedure in the experiments, where we can see both the difference between reward functions and the difference between returns from the learned policies. Though I’m not sure what additional information the Highway domain gives compared to the Objectworld.   On line 271, k was chosen using domain knowledge. Are there any guidelines for choosing k in arbitrary domains? What was the value of k for these experiments?  Please cite which paper GPIRL comes from on line 276. I don’t remember seeing this acronym in the paper (even on line 48).    Minor/Typos:  In Algorithm 1, I believe the $i$ should be a $j$ (or vice versa)