summaries: This paper aims to address the problem of lacking sufficient demonstrations in inverse reinforcement learning (IRL) problems. They propose to take a meta learning approach, in which a set of i.i.d. IRL tasks are provided to the learner and the learner aims to learn a strategy to quickly recover a good reward function for a new task that is assumed to be sampled from the same task distribution. Particularly, they adopt the gradient-based meta learning algorithm, MAML, and the maximal entropy (MaxEnt) IRL framework, and derive the required meta gradient expression for parameter update. The proposed algorithm is evaluated on a synthetic grid-world problem, SpriteWorld. The experimental results suggest the proposed algorithm can learn to mimic the optimal policy under the true reward function that is unknown to the learner.  This paper attempts to the solve  data-set coverage issue common with Inverse reinforcement learning based approaches - by introducing a meta-learning framework trained on a smaller number of basic tasks. The primary insight here is that there exists a smaller set of unique tasks, the knowledge from which is transferable to new tasks and using these to learn an initial parametrized reward function improves the coverage for IRL. With experiments on the SpriteWorld synthetic data-set, the authors confirm this hypothesis and demonstrate performance benefits - showcasing better correlation with far fewer  number of demonstrations.
abstractText: A significant challenge for the practical application of reinforcement learning to real world problems is the need to specify an oracle reward function that correctly defines a task. Inverse reinforcement learning (IRL) seeks to avoid this challenge by instead inferring a reward function from expert behavior. While appealing, it can be impractically expensive to collect datasets of demonstrations that cover the variation common in the real world (e.g. opening any type of door). Thus in practice, IRL must commonly be performed with only a limited set of demonstrations where it can be exceedingly difficult to unambiguously recover a reward function. In this work, we exploit the insight that demonstrations from other tasks can be used to constrain the set of possible reward functions by learning a “prior” that is specifically optimized for the ability to infer expressive reward functions from limited numbers of demonstrations. We demonstrate that our method can efficiently recover rewards from images for novel tasks and provide intuition as to how our approach is analogous to learning a prior.
references.authors: Pieter Abbeel Andrew Y. Ng
references.title: Apprenticeship learning via inverse reinforcement learning
references.venue: In Proceedings of the Twenty-first International Conference on Machine Learning,
references.authors: Dario Amodei Chris Olah Jacob Steinhardt Paul Christiano John Schulman Dan Mané
references.title: Concrete problems in ai safety
references.venue: arXiv preprint arXiv:1606.06565,
references.authors: Marcin Andrychowicz Misha Denil Sergio Gomez Matthew W Hoffman David Pfau Tom Schaul Nando de Freitas
references.title: Learning to learn by gradient descent by gradient descent
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Monica Babeş-Vroman Vukosi Marivate Kaushik Subramanian Michael Littman
references.title: Apprenticeship learning about multiple intentions
references.venue: In International Conference on International Conference on Machine Learning,
references.authors: Chris Baker Joshua B Tenenbaum Rebecca R Saxe
references.title: Goal inference as inverse planning
references.authors: Jaedeug Choi Kee-Eung Kim
references.title: Nonparametric bayesian inverse reinforcement learning for multiple reward functions
references.venue: In Advances in Neural Information Processing Systems, NIPS,
references.authors: Christos Dimitrakakis Constantin A. Rothkopf
references.title: Bayesian multitask inverse reinforcement learning
references.venue: In European Conference on Recent Advances in Reinforcement Learning, EWRL,
references.authors: Yan Duan John Schulman Xi Chen Peter L Bartlett Ilya Sutskever Pieter Abbeel
references.title: Rl2: Fast reinforcement learning via slow reinforcement learning
references.venue: arXiv preprint arXiv:1611.02779,
references.authors: Yan Duan Marcin Andrychowicz Bradly C. Stadie Jonathan Ho Jonas Schneider Ilya Sutskever Pieter Abbeel Wojciech Zaremba
references.title: One-shot imitation learning
references.venue: In NIPS,
references.authors: David Duvenaud Dougal Maclaurin Ryan Adams
references.title: Early stopping as nonparametric variational inference
references.venue: In Artificial Intelligence and Statistics,
references.authors: Chelsea Finn Pieter Abbeel Sergey Levine
references.title: Model-agnostic meta-learning for fast adaptation of deep networks
references.venue: In International Conference on Machine Learning,
references.authors: Chelsea Finn Tianhe Yu Tianhao Zhang Pieter Abbeel Sergey Levine
references.title: One-shot visual imitation learning via meta-learning
references.venue: arXiv preprint arXiv:1709.04905,
references.authors: Justin Fu Katie Luo Sergey Levine
references.title: Learning robust rewards with adverserial inverse reinforcement learning
references.venue: International Conference on Learning Representations,
references.authors: Erin Grant Chelsea Finn Sergey Levine Trevor Darrell Thomas Griffiths
references.title: Recasting gradientbased meta-learning as hierarchical bayes
references.venue: International Conference on Learning Representations (ICLR),
references.authors: Karol Hausman Yevgen Chebotar Stefan Schaal Gaurav Sukhatme Joseph J Lim
references.title: Multimodal imitation learning from unstructured demonstrations using generative adversarial nets
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Jonathan Ho Stefano Ermon
references.title: Generative adversarial imitation learning
references.venue: In Neural Information Processing Systems (NIPS),
references.authors: Sepp Hochreiter Jürgen Schmidhuber
references.title: Long short-term memory
references.venue: Neural computation,
references.authors: Diederik P Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: arXiv preprint arXiv:1412.6980,
references.authors: Jens Kober Jan Peters
references.title: Reinforcement learning in robotics: A survey
references.venue: In Reinforcement Learning,
references.authors: Gregory Koch
references.title: Siamese neural networks for one-shot image recognition
references.authors: Alex Kuefler Mykel J. Kochenderfer
references.title: Burn-in demonstrations for multi-modal imitation learning
references.authors: Sergey Levine Zoran Popovic Vladlen Koltun
references.title: Nonlinear inverse reinforcement learning with gaussian processes
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Sergey Levine Chelsea Finn Trevor Darrell Pieter Abbeel
references.title: End-to-end training of deep visuomotor policies
references.venue: Journal of Machine Learning Research (JMLR),
references.authors: Ke Li Jitendra Malik
references.title: Learning to optimize neural nets
references.venue: arXiv preprint arXiv:1703.00441,
references.authors: Kun Li Joel W. Burdick
references.title: Meta inverse reinforcement learning via maximum reward sharing for human motion analysis
references.authors: Yunzhu Li Jiaming Song Stefano Ermon
references.title: Inferring the latent structure of human decisionmaking from raw visual inputs
references.venue: arXiv preprint arXiv:1703.08840,
references.authors: Nikhil Mishra Mostafa Rohaninejad Xi Chen Pieter Abbeel
references.title: Meta-learning with temporal convolutions
references.venue: arXiv preprint arXiv:1707.03141,
references.authors: Volodymyr Mnih Koray Kavukcuoglu David Silver Andrei A Rusu Joel Veness Marc G Bellemare Alex Graves Martin Riedmiller Andreas K Fidjeland Georg Ostrovski
references.title: Human-level control through deep reinforcement learning
references.venue: Nature, 518(7540):529–533,
references.authors: Igor Mordatch
references.title: Concept learning with energy-based models
references.venue: In ICLR Workshop,
references.authors: Devang K Naik RJ Mammone
references.title: Meta-neural networks that learn by learning
references.venue: In Neural Networks,
references.authors: Andrew Y. Ng Stuart J. Russell
references.title: Algorithms for inverse reinforcement learning
references.venue: In Proceedings of the Seventeenth International Conference on Machine Learning,
references.authors: Andrew Y Ng Daishi Harada Stuart Russell
references.title: Policy invariance under reward transformations: Theory and application to reward shaping
references.authors: Deepak Ramachandran Eyal Amir
references.title: Bayesian inverse reinforcement learning
references.venue: In International Joint Conference on Artifical Intelligence,
references.authors: Nathan D Ratliff J Andrew Bagnell Martin A Zinkevich
references.title: Maximum margin planning
references.venue: In Proceedings of the 23rd international conference on Machine learning,
references.authors: Sachin Ravi Hugo Larochelle
references.title: Optimization as a model for few-shot learning
references.authors: Scott Reed Yutian Chen Thomas Paine Aaron van den Oord Oriol Vinyals SM Ali Eslami Danilo Rezende Nando de Freitas
references.title: Few-shot autoregressive density estimation: Towards learning to learn distributions
references.venue: In ICLR,
references.authors: Danilo Jimenez Rezende Shakir Mohamed Ivo Danihelka Karol Gregor Daan Wierstra
references.title: Oneshot generalization in deep generative models
references.venue: arXiv preprint arXiv:1603.05106,
references.authors: Adam Santoro Sergey Bartunov Matthew Botvinick Daan Wierstra Timothy Lillicrap
references.title: Metalearning with memory-augmented neural networks
references.venue: In International conference on machine learning,
references.authors: Reginaldo J Santos
references.title: Equivalence of regularization and truncated iteration for general ill-posed problems
references.venue: Linear algebra and its applications,
references.authors: Jürgen Schmidhuber
references.title: Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-.
references.venue: hook. PhD thesis, Technische Universität München,
references.authors: E. Shelhamer J. Long T. Darrell
references.title: Fully convolutional networks for semantic segmentation
references.venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,
references.authors: Pranav Shyam Shubham Gupta Ambedkar Dukkipati
references.title: Attentive recurrent comparators
references.venue: arXiv preprint arXiv:1703.00767,
references.authors: Hava T Siegelmann Eduardo D Sontag
references.title: On the computational power of neural nets
references.venue: Journal of computer and system sciences,
references.authors: David Silver Aja Huang Chris J Maddison Arthur Guez Laurent Sifre George Van Den Driessche Julian Schrittwieser Ioannis Antonoglou Veda Panneershelvam Marc Lanctot
references.title: Mastering the game of go with deep neural networks and tree
references.venue: search. Nature,
references.authors: Jonas Sjöberg Lennart Ljung
references.title: Overtraining, regularization and searching for a minimum, with application to neural networks
references.venue: International Journal of Control,
references.authors: Jake Snell Kevin Swersky Richard Zemel
references.title: Prototypical networks for few-shot learning
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Gabriel Synnaeve Nantas Nardelli Alex Auvolat Soumith Chintala Timothée Lacroix Zeming Lin Florian Richoux Nicolas Usunier
references.title: Torchcraft: a library for machine learning research on real-time strategy games
references.venue: arXiv preprint arXiv:1611.00625,
references.authors: Sebastian Thrun Lorien Pratt
references.title: Learning to learn
references.venue: Springer Science & Business Media,
references.authors: Oriol Vinyals Charles Blundell Tim Lillicrap Daan Wierstra
references.title: Matching networks for one shot learning
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Jane X Wang Zeb Kurth-Nelson Dhruva Tirumala Hubert Soyer Joel Z Leibo Remi Munos Charles Blundell Dharshan Kumaran Matt Botvinick
references.title: Learning to reinforcement learn
references.venue: arXiv preprint arXiv:1611.05763,
references.authors: M. Wulfmeier D.Z. Wang I. Posner
references.title: Watch this: Scalable cost-function learning for path planning in urban environments
references.venue: In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS),
references.authors: Markus Wulfmeier Peter Ondruska Ingmar Posner
references.title: Maximum entropy deep inverse reinforcement learning
references.venue: In Neural Information Processing Systems Conference, Deep Reinforcement Learning Workshop, volume abs/1507.04888,
references.authors: Markus Wulfmeier Dushyant Rao Ingmar Posner
references.title: Incorporating human domain knowledge into large scale cost function learning
references.venue: CoRR, abs/1612.04318,
references.authors: Wojciech Zaremba Ilya Sutskever Oriol Vinyals
references.title: Recurrent neural network regularization
references.venue: arXiv preprint arXiv:1409.2329,
references.authors: Brian D. Ziebart Andrew Maas J. Andrew Bagnell Anind K. Dey
references.title: Maximum entropy inverse reinforcement learning
references.venue: In Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 3,
sections.heading: 1 INTRODUCTION
sections.text: Reinforcement learning (RL) algorithms have the potential to automate a wide range of decisionmaking and control tasks across a variety of different domains, as demonstrated by successful recent applications ranging from robotic control (Kober & Peters, 2012; Levine et al., 2016) to game playing (Mnih et al., 2015; Silver et al., 2016). A key assumption of the RL problem statement is the availability of a reward function that accurately describes the desired tasks. For many real world tasks, reward functions can be challenging to manually specify, while being crucial to good performance (Amodei et al., 2016). Most real world tasks are multifaceted and require reasoning over multiple factors in a task (e.g. an autonomous vehicle navigating a city at night), while simultaneously providing appropriate reward shaping to make the task feasible with tractable exploration (Ng et al., 1999). These challenges are compounded by the inherent difficulty of specifying rewards for tasks with high-dimensional observation spaces such as images.
  Inverse reinforcement learning (IRL) is an approach that aims to address this problem by instead inferring the reward function from demonstrations of the task (Ng & Russell, 2000). This has the appealing benefit of taking a data-driven approach to reward specification in place of hand engineering. In practice however, rewards functions are rarely learned as it can be prohibitively expensive to provide demonstrations that cover the variability common in real world tasks (e.g., collecting demonstrations of opening every type of door knob). In addition, while learning a complex function from high dimensional observations might make an expressive function approximator seem like a reasonable modelling assumption, in the “few-shot” domain it is notoriously difficult to unambiguously recover a good reward function with expressive function approximators. Prior solutions have thus instead relied on low-dimensional linear models with handcrafted features that effectively encode a strong prior on the relevant features of a task. This requires engineering a set of features by hand that work well for a specific problem. In this work, we propose an approach that instead explicitly learns expressive features that are robust even when learning with limited demonstrations.
  Our approach relies on the key observation that related tasks share common structure that we can leverage when learning new tasks. To illustrate, considering a robot navigating through a home. While the exact reward function we provide to the robot may differ depending on the task, there is a structure amid the space of useful behaviours, such as navigating to a series of landmarks, and
  there are certain behaviors we always want to encourage or discourage, such as avoiding obstacles or staying a reasonable distance from humans. This notion agrees with our understanding of why humans can easily infer the intents and goals (i.e., reward functions) of even abstract agents from just one or a few demonstrations Baker et al. (2007), as humans have access to strong priors about how other humans accomplish similar tasks accrued over many years. Similarly, our objective is to discover the common structure among different tasks, and encode the structure in a way that can be used to infer reward functions from a few demonstrations.
  More specifically, in this work we assume access to a set of tasks, along with demonstrations of the desired behaviors for those tasks, which we refer to as the meta-training set. From these tasks, we then learn a reward function parameterization that enables effective few-shot learning when used to initialize IRL in a novel task. Our method is summarized in Fig. 1. Our key contribution is an algorithm that enables efficient learning of new reward functions by using meta-training to build a rich “prior” for goal inference. Using our proposed approach, we show that we
  can learn deep neural network reward functions from raw pixel observations with substantially better data efficiency than existing methods and standard baselines.
sections.heading: 2 RELATED WORK
sections.text: Inverse reinforcement learning (IRL) (Ng & Russell, 2000) is the problem of inferring an expert’s reward function directly from demonstrations. Prior methods for performing IRL range from margin based approaches (Abbeel & Ng, 2004; Ratliff et al., 2006) to probabilistic approaches (Ramachandran & Amir, 2007; Ziebart et al., 2008). Although it is possible to extend our approach to any other IRL method, in this work we base on work on the maximum entropy (MaxEnt) framework (Ziebart et al., 2008). In addition to allowing for sub-optimality in the expert demonstrations, MaxEnt-IRL can be re-framed as a maximum likelihood estimation problem. 4).
  In part to combat the under-specified nature of IRL, prior work has often used low-dimensional linear parameterizations with handcrafted features (Abbeel & Ng, 2004; Ziebart et al., 2008). In order to learn from high dimensional input, Wulfmeier et al. (2015) proposed applying fully convolutional networks (Shelhamer et al., 2017) to the MaxEnt IRL framework (Ziebart et al., 2008) for several navigation tasks (Wulfmeier et al., 2016a;b). Other methods that have incorporated neural network rewards include guided cost learning (GCL) (Finn et al., 2017a), which uses importance sampling and regularization for scalability to high-dimensional spaces, and adversarial IRL (Fu et al., 2018). Several other methods have also proposed imitation learning approaches based on adversarial frameworks that resemble IRL, but do not aim to directly recover a reward function (Ho & Ermon, 2016; Li et al., 2017; Hausman et al., 2017; Kuefler & Kochenderfer, 2018). In this work, instead of improving the ability to learn reward functions on a single task, we focus on the problem of effectively learning to use prior demonstration data from other IRL tasks, allowing us to learn new tasks from a limited number demonstrations even with expressive non-linear reward functions.
  Prior work has explored the problem of multi-task IRL, where the demonstrated behavior is assumed to have originated from multiple experts achieving different goals. Some of these approaches include those that aim to incorporate a shared prior over reward functions through extending the Bayesian
  IRL (Ramachandran & Amir, 2007) framework to the multi-task setting (Dimitrakakis & Rothkopf, 2012; Choi & Kim, 2012). Other approaches have clustered demonstrations while simultaneously inferring reward functions for each cluster (Babeş-Vroman et al., 2011) or introduced regularization between rewards to a common “shared reward” (Li & Burdick, 2017). Our work is similar in that we also seek to encode prior information common to the tasks. However, a critical difference is that our method specifically aims to distill the meta-training tasks into a prior that can then be used to learn rewards for new tasks efficiently. The goal therefore is not to acquire good reward functions that explain the meta-training tasks, but rather to use them to learn efficiently on new tasks.
  Our approach builds on work on the broader problem of meta-learning (Schmidhuber, 1987; Bengio et al.; Naik & Mammone, 1992; Thrun & Pratt, 2012) and generative modelling (Rezende et al., 2016; Reed et al., 2018; Mordatch, 2018). Prior work has proposed a variety of solutions for learning to learn including memory based methods (Duan et al., 2016; Santoro et al., 2016; Wang et al., 2016; Mishra et al., 2017), methods that learn an optimizer and/or initialization (Andrychowicz et al., 2016; Ravi & Larochelle, 2016; Finn et al., 2017a; Li & Malik, 2017), and methods that compare new datapoints in a learned metric space (Koch, 2015; Vinyals et al., 2016; Shyam et al., 2017; Snell et al., 2017). Our work is motivated by the goal of broadening the applicability of IRL, but in principle it is possible to adapt many of these meta-learning approaches for our problem statement. We leave it to future work to do a comprehensive investigation of different meta-learning approaches which could broaden the applicability of IRL.
sections.heading: 3 PRELIMINARIES AND OVERVIEW
sections.text: In this section, we introduce our notation and describe the IRL and meta-learning problems.
sections.heading: 3.1 LEARNING REWARDS VIA INVERSE REINFORCEMENT LEARNING
sections.text: The standard Markov decision process (MDP) is defined by the tuple (S,A, ps, r, γ) where S and A denote the set of possible states and actions respectively, r : S × A → R is the reward function, γ ∈ [0, 1] is the discount factor and ps : S ×S ×A → [0, 1] denotes the transition distribution over the next state st+1, given the current state st and current action at. Typically, the goal of “forward” RL is to maximize the expected discounted return R(τ) = ∑T t=1 γ t−1r(st,at).
  In IRL, we instead assume that the reward function is unknown but that we instead have access to a set of expert demonstrations D = {τ1, . . . , τK}, where τk = {s1,a1, . . . , sT ,aT }. The goal of IRL is to recover the unknown reward function r from the set of demonstrations. We build on the maximum entropy (MaxEnt) IRL framework by Ziebart et al. (2008), which models the probability of the trajectories as being distributed proportional to their exponentiated return
  p(τ) = 1
  Z exp (R(τ)) , (1) where Z is the partition function, Z = ∫ τ exp(R(τ))dτ . This distribution can be shown to be induced by the optimal policy in entropy regularized forward RL problem:
  π∗ =π Eτ∼π [R(τ)− log π(τ)] . (2) This formulation allows us to pose the reward learning problem as a maximum likelihood estimation (MLE) problem in an energy-based model rφ:
  min φ Eτ∼D [LIRL(τ)]=min φ Eτ∼D [− log pφ(τ)] . (3)
  Learning in general energy-based models of this form is common in many applications such as structured prediction. However, in contrast to applications where learning can be supervised by millions of labels (e.g. semantic segmentation), the learning problem in Eq. 3 must typically be performed with a relatively small number of example demonstrations. In this work, we seek to address this issue in IRL by providing a way to integrate information from prior tasks to constrain the optimization in Eq. 3 in the regime of limited demonstrations.
sections.heading: 3.2 META-LEARNING
sections.text: The goal of meta-learning algorithms is to optimize for the ability to learn efficiently on new tasks. Rather than attempt to generalize to new datapoints, meta-learning can be understood as attempting
  to generalize to new tasks. It is assumed in the meta-learning setting that there are two sets of tasks that we refer to as the meta-training set {Ti ; i = 1..N} and meta-test set {Tj ; j = 1..M}, which are both drawn from a distribution p(T ). During meta-training time, the meta-learner attempts to learn the structure of the tasks in the meta-training set, such that when it is presented with a new test task, it can leverage this structure to learn efficiently from a limited number of examples.
  To illustrate this distinction, consider the case of few-shot learning setting. Let fθ denote the learner, and let a task be defined by learning from K training examples X trT = {x1 . . . ,xK}, Y trT = {y1 . . . ,yK}, and evaluating on K ′ test examples X testT = {x1 . . . ,xK′}, Y testT = {y1 . . . ,yK′}. One approach to meta-learning is to directly parameterize the meta-learner with an expressive model such as a recurrent or recursive neural network (Duan et al., 2016; Mishra et al., 2017) conditioned on the task training data and the inputs for the test task: fθ(Y |X testT , X trT , Y trT ). Such a model is optimized using log-likelihood across all tasks. In this approach to meta-learning, since neural networks are known to be universal function approximators (Siegelmann & Sontag, 1995), any desired structure between tasks can be implicitly encoded.
  Rather than learn a single black-box function, another approach to meta-learning is to learn components of the learning procedure such as the initialization (Finn et al., 2017a) or the optimization algorithm (Ravi & Larochelle, 2016; Andrychowicz et al., 2016). In this work we extend the approach of model agnostic meta-learning (MAML) introduced by Finn et al. (2017a), which learns an initialization that is adapted by gradient descent. Concretely, in the supervised learning case, given a loss function L(θ, XT , YT ) (e.g. cross-entropy), MAML performs the following optimization
  min θ ∑ T L(φT , X testT , Y testT ) = min θ ∑ T L ( θ − α∇θL(θ, X trT , Y trT ), X testT , Y testT ) , (4)
  where the optimization is over an initial set of parameters θ and the loss on the held out tasks XtestT becomes the signal for learning the initial parameters for gradient descent on X tr T . This optimization is analogous to adding a constraint in a multi-task setting, which we show in later sections is analogous in our setting to learning a prior over reward functions.
sections.heading: 4 LEARNING TO LEARN REWARDS
sections.text: Our goal in meta-IRL is to learn how to learn reward functions across many tasks such that the model can infer the reward function for a new task using only one or a few expert demonstrations. Intuitively, we can view this problem as aiming to learn a prior over the intentions of human demonstrators, such that when given just one or a few demonstrations of a new task, we can combine the learned prior with the new data to effectively determine the human’s reward function. Such a prior is helpful in inverse reinforcement learning settings, since the space of relevant reward functions is much smaller than the space of all possible rewards definable on the raw observations.
  During meta-training, we have a set of tasks {Ti ; i = 1..N}. Each task Ti has a set of demonstrations DT = {τ1, . . . , τK} from an expert policy which we partition into disjoint DtrT and DtestT sets. The demonstrations for each meta-training task are assumed to be produced by the expert according to the maximum entropy model in Section 3.1. During meta-training, these tasks will be used to encodes common structure so that our model can quickly acquire rewards for new tasks from just a few demonstrations.
  After meta-training, our method is presented with a new task. During this meta-test phase, the algorithm must infer the parameters of the reward function rφ(st,at) for the new task from a few demonstrations. As is standard in meta-learning, we assume that the test task is from the same distribution of tasks seen during meta-training, a distribution that we denote as p(T ).
sections.heading: 4.1 META REWARD AND INTENTION LEARNING (MANDRIL)
sections.text: In order to meta-learn a reward function that can act as a prior for new tasks and new environments, we first formalize the notion of a good reward by defining a loss LT (θ) on the reward function rθ for a particular task T . We use the MaxEnt IRL loss LIRL discussed in Section 3, which, for a given DT , leads to the following gradient (Ziebart et al., 2008):
  ∇θLT (θ) = ∂rθ ∂θ [ Eτ [µτ ]− µDT ] . (5)
  Algorithm 1 Meta Reward and Intention Learning (MandRIL)
  1: Input: Set of meta-training tasks {T }meta-train 2: Input: hyperparameters α, β 3: function MAXENTIRL-GRAD(rθ , T , D) . Single task update 4: µD = STATE-VISITATIONS-TRAJ(T , D) . Compute state visitations of demos 5: Eτ [µτ ] = STATE-VISITATIONS-POLICY(rθ , T ) . Compute Max-Ent state visitations 6: ∂L
  ∂rθ = Eτ [µτ ]− µD . MaxEntIRL gradient (Ziebart et al., 2008)
  7: Return ∂L ∂rθ 8: 9: Randomly initialize θ
  10: while not done do 11: Sample batch of tasks Ti ∼ {T }meta-train 12: for all Ti do 13: Sample demos Dtr = {τ1, . . . , τK} ∼ Ti 14:
  ∂L trTi (θ) ∂rθ = MAXENTIRL-GRAD(rθ , Ti, Dtr) . Inner loss computation
  15: Compute∇θL trTi(θ) from ∂L trTi (θ) ∂rθ
  via chain rule 16: Compute updated parameters φTi = θ − α∇θL tr Ti(θ) . Fast update 17: Sample demos Dtest = {τ ′1, . . . , τ ′K′} ∼ Ti 18:
  ∂L testTi ∂rθ = MAXENTIRL-GRAD(rφTi , Ti, D test)) . Outer loss computation
  19: Compute∇θL testTi from ∂L testTi ∂rθ via chain rule . Compute meta-gradient 20: Compute update to θ ← θ − β ∑ i∇θL test Ti . Update initial parameters
  where µτ are the state visitations under the optimal maximum entropy policy under rθ, and µD are the mean state visitations under the demonstrated trajectories.
  If our end goal were to achieve a single reward function that works as well as possible across all tasks in {Ti ; i = 1..N}, then we could simply follow the mean gradient across all tasks. However, our objective is different: instead of optimizing performance on the meta-training tasks, we aim to learn a reward function that can be quickly and efficiently adapted to new tasks at meta-test time. In doing so, we aim to encode prior information over the task distribution in this learned reward prior.
  We propose to implement such a learning algorithm by finding the parameters θ, such that starting from θ and taking a small number of gradient steps on a few demonstrations from given task leads to a reward function for which a set of test demonstrations have high likelihood, with respect to the MaxEnt IRL model. In particular, we would like to find a θ such that the parameters
  φT = θ − α∇θL trT (θ) (6)
  lead to a reward function rφT for task T , such that the IRL loss (corresponding to negative loglikelihood) for a disjoint set of test demonstrations, given by LT ,testIRL , is minimized. The corresponding optimization problem for θ can therefore be written as follows:
  min θ N∑ i=1 L testTi (φTi) = N∑ i=1 L testTi ( θ − α∇θL trTi(θ) ) . (7)
  Our method acquires this prior θ over rewards in the task distribution p(T ) by optimizing this loss. This amounts to an extension of the MAML algorithm in Section 3.2 to the inverse reinforcement learning setting. This extension is quite challenging, because computing the MaxEnt IRL gradient requires repeatedly solving for the current maximum entropy policy and visitation frequencies, and the MAML objective requires computing derivatives through this gradient step. Next, we describe in detail how this is done. An overview of our method is also outlined in Alg. 1.
  Meta-training. The computation of the meta-gradient for the objective in Eq. 7 can be conceptually separated into two parts. First, we perform the update in Eq. 6 by computing the expected state visitations µ, which is the expected number of times an agent will visit each state. We denote this overall procedure as STATE-VISITATIONS-POLICY, and follow Ziebart et al. (2008) by first computing the maximum entropy optimal policy in Eq. 2 under the current rθ, and then approximating µ
  using dynamic programming. Next, we compute the state visitation distribution of the expert using a procedure which we denote as STATE-VISITATIONS-TRAJ. This can be done either empirically, by averaging the state visitation of the experts demonstrations, or by using STATE-VISITATIONSPOLICY if the true reward is available at meta-training time. This allows us to recover the IRL gradient according to Eq. 5, which we can then apply to compute φT according to Eq. 6.
  Second, we need to differentiate through this update to compute the gradient of the meta-loss in Eq. 7. Note that the meta-loss itself is the IRL loss evaluated with a different set of test demonstrations. We follow the same procedure as above to evaluate the gradient of LT ,testIRL with respect to the post-update parameters φT , and then apply the chain rule to compute the meta-gradient:
  ∇θL testT (θ) = ∂L testT ∂rφT ∂rφT ∂φT ∂ ∂θ (θ − α∇θL trT (θ))
  = ∂L testT ∂rφT ∂rφT ∂φT
  ( I− α∂
  2L trT (θ) ∂θ2 − α∂ rθ ∂ θ ∂ rθ ∂ θ > ∂ ∂ rθ Eτ [µτ ]
  ) (8)
  where on the second line we differentiate through the MaxEnt-IRL update. The derivation of this expression is somewhat more involved and provided in the supplementary Appendix D.
  Meta-testing. Once we have acquired the meta-trained parameters θ that encode a prior over p(T ), we can leverage this prior to enable fast, few-shot IRL of novel tasks in {Tj ; j = 1..M}. For each task, we first compute the state visitations from the available set of demonstrations for that task. Next, we use these state visitations to compute the gradient, which is the same as the inner loss gradient computation of the meta-training loop in Alg. 1. We apply this gradient to adapt the parameters θ to the new task. Even if the model was trained with only one to three inner gradient steps, we found in practice that it was beneficial to take substantially more gradient steps during meta-testing; performance continued to improve with up to 20 steps.
sections.heading: 4.2 INTERPRETATION AS LEARNING A PRIOR OVER INTENT
sections.text: The objective in Eq. 6 optimizes for parameters that enable that reward function to adapt and generalize efficiently on a wide range of tasks. Intuitively, constraining the space of reward functions to lie within a few steps of gradient descent can be interpreted as expressing a “locality” prior over reward function parameters. This intuition can be made more concrete with the following analysis.
  By viewing IRL as maximum likelihood estimation, we can take the perspective of Grant et al. (2018) who showed that for a linear model, fast adaptation via a few steps of gradient descent in MAML is performing MAP inference over φ, under a Gaussian prior with the mean θ and a covariance that depends on the step size, number of steps and curvature of the loss. This is based on the connection between early stopping and regularization previously discussed in Santos (1996), which we refer the readers to for a more detailed discussion. The interpretation of MAML as imposing a Gaussian prior on the parameters is exact in the case of a likelihood that is quadratic in the parameters (such as the log-likelihood of a Gaussian in terms of its mean). For any non-quadratic likelihood, this is an approximation in a local neighborhood around θ (i.e. up to convex quadratic approximation). In the case of very complex parameterizations, such
  as deep function approximators, this is a coarse approximation and unlikely to be the mode of a posterior. However, we can still frame the effect of early stopping and initialization as serving as a prior in a similar way as prior work (Sjöberg & Ljung, 1995; Duvenaud et al., 2016; Grant et al., 2018). More importantly, this interpretation hints at future extensions to our approach that could benefit from employing more fully Bayesian approaches to reward and goal inference.
sections.heading: 5 EXPERIMENTS
sections.text: Our evaluation seeks to answer two questions. First, we aim to test our core hypothesis that leveraging prior task experience enables reward learning for new tasks with just a few demonstrations. Second, we compare our method with alternative algorithms that make use of multi-task experience.
  We test our core hypothesis by comparing learning performance on a new task starting from the learned initialization produced by MandRIL, compared to starting from scratch with a random initialization. This comparison is meant to evaluate whether prior experience on other tasks can in fact make inverse RL more efficient.
  To our knowledge, there is no prior work that addresses the meta-inverse reinforcement learning problem introduced in this paper. Thus, to provide a point of comparison and calibrate the difficulty of the tasks, we adapt two alternative black-box meta-learning methods to the IRL setting. The comparisons to both of the black-box methods described below evaluate the importance of incorporating the IRL gradient into the meta-learning process, rather than learning the adaptation process entirely from scratch.
  • Demo conditional model: Our method implicitly conditions on the demonstrations through the gradient update. In principle, a conditional deep model with sufficient capacity could implicitly implement a similar learning rule. Thus, we consider a conditional model (often referred to as a “contextual model” (Finn et al., 2017b)), which receives the demonstration as an additional input.
  • Recurrent meta-learner: We additionally compare to an RNN-based meta-learner (Santoro et al., 2016; Duan et al., 2017). Specifically, we implement a conditional model by feeding both images and sequences of states visited by the demonstrations to an LSTM.
  Our approach can be understood as explicitly optimizing for an effective parameter initialization for the IRL problem. In order to test the benefits of our proposed formulation, we also compare with finetuning an initialization obtained with the same set of prior tasks, but with supervised pretraining as follows:
  • Supervised pre-training: We compare to following the average gradient during meta-training, averaged across tasks, and fine-tuning at meta-test time (as discussed in Section 4). This comparison evaluates the benefits of optimizing explicitly for weights that perform well under fine-tuning. We compare to pre-training on a single task as well as all the meta-training tasks.
  Next, we describe our environment and evaluation.
  Spriteworld navigation domain. Since most prior IRL works (and multi-task IRL works) have studied settings where linear reward function approximators suffice (i.e. low-dimensional state spaces made up from hand-designed features), we design an experiment that is significantly more challenging—that requires learning rewards on raw pixels—while still exhibiting multi-task structure needed to test our core hypothesis. We consider a navigation problem where we aim to learn a convolutional neural network that directly maps image pixels to rewards. To do so, we introduce “SpriteWorld,” which is a synthetically generated task, some examples of which are shown in Fig. 3. The task visuals are inspired by Starcraft and work applying learning algorithms to perform micromanagement (e.g. (Synnaeve et al., 2016)), although we do not use the game engine. Tasks involve navi-
  gating to goal objects while exhibiting preference over terrain types (e.g. the agent prefers to traverse dirt tiles over traversing grass tiles). At meta-test time, we provide one or a few demonstrations in a single training environment and evaluate the reward learned using these demonstrations in a new, test environment that contains the same objects as the training environment, but arranged differently. Evaluating in a new test environment is critical to measure that the reward learned the correct visual cues, rather than simply memorizing the demonstration trajectory.
  The underlying MDP structure of SpriteWorld is a grid, where the states are each of the grid cells, and the actions enable the agent to move to any one of its 8-connected neighbors. We generate unique tasks from this domain as follows. First, we randomly choose a set of 3 sprites from a total of 100 sprites from the original game (creating a total of 161,700 unique tasks). We randomly place these three sprites within a randomly generated terrain tiling; we designate one of the sprites to be the goal of the navigation task, i.e. the object to which the agent must navigate. The other two objects are treated as obstacles for which the agent incurs a large negative reward for not avoiding. In each task, we optimize our model on a meta-training set and evaluate the ability of the reward function to the generalize to a rearrangement of the same objects. For example, suppose the goal of the task is to navigate to sprite A, while avoiding sprites B and C. Then, to generate an environment for evaluation, we resample the positions of the sprites, while the underlying task remains the same (i.e., navigate to A). This requires the model to make use of the visual patterns in the scene to generalize effectively, rather than simply memorizing positions of sprites. We evaluate on novel combinations of units seen in meta-training, as well as the ability to generalize to new unseen units. We provide further details on our setup in Appendices A and B.
  We measure performance using the expected value difference, which measures the sub-optimality of a policy learned under the learned reward; this is a standard performance metric used in prior IRL work (Levine et al., 2011; Wulfmeier et al., 2015). The metric is computed by taking the difference between the value of the optimal policy under the
  learned reward and the value of the optimal policy under the true reward.
  Evaluation protocol. We evaluate on held-out tasks that were unseen during meta-training. We consider two settings: (1) tasks involving new combinations and placements of sprites, with individual sprites that were present during meta-training, and (2) tasks with combinations of entirely new sprites which we refer to as “out of domain objects.” For each task, we generate one environment (set of sprite positions) along with one or a few demonstrations for adapting the reward, and generate a second environment (with new sprite positions) where we evaluate the adapted reward. In these meta-test time evaluations, we refer to the performance on the first environment as “training performance” (not to be confused with meta-training) and to performance on the second as “testing performance”. We evaluate on 32 meta-test randomly generated tasks.
  Results. The results are shown in Fig. 4, which illustrate test performance with in-distribution and out-of-distribution sprites. Our approach, MandRIL, achieves consistently better performance in both settings. Most significantly, our approach performs well even with single-digit numbers of demonstrations. By comparison, alternative meta-learning methods generally overfit considerably, attaining good training performance (see Appendix. A for curves) but poor test performance. Learning the reward function from scratch is in fact the most competitive baseline – as the number of demonstrations increases, simply training the fully convolutional reward function from scratch on the new task is the only method that matches the performance of MandRIL when provided 20 or more demonstrations. However, with only a few demonstrations, MandRIL has substantially lower value difference. It is worth noting the performance of MandRIL on the out of distribution test setting (Fig. 4, bottom): although the evaluation is on new sprites, MandRIL is still able to adapt via gradient descent and exceed the performance of learning from scratch and all other methods.
  Finally, as it is common practice to fine-tune representations obtained from a supervised pre-training phase, we perform this comparison in Figure 5. We compare against an approach that follows the mean gradient across the tasks at meta-training time and is fine-tuned at meta-test time which we find is less effective than learning from a random initialization. We conclude that fine tuning reward functions learned in this manner is not an effective way of using prior task information. When using a single task for pre-training, we find that it performs comparable to random initialization. In contrast, we find that our approach, which explicitly optimizes for initial weights for fine-tuning, robustly improves performance.
sections.heading: 6 CONCLUSION
sections.text: In this work, we present an approach that enables few-shot learning for reward functions of new tasks. We achieve this through a novel formulation of inverse reinforcement learning that learns to encode common structure across tasks. Using our meta-IRL approach, we show that we can leverage data from previous tasks to effectively learn deep neural network reward functions from raw pixel observations for new tasks, from only a handful of demonstrations. Our work paves the way for futures work that considers environments with unknown dynamics, or more fully probabilistic approaches to reward and goal inference.
sections.heading: A EXPERIMENTAL DETAILS
sections.text: The input to our reward function for all experiments is a resized 80×80 RGB image, with an output space of 20×20 in the underlying MDP state space s. In order to compute the optimal policy, we use Q-iteration. In our experiments, we parameterize the reward function for all our reward functions starting from the same base learner. The first layer is a 8 × 8 convolution with a stride of 2, 256 filters and symmetric padding of 4. The second layer is a 4 × 4 with a stride of 2, 128 filters and symmetric of 1. The third and fourth layer are 3 × 3 convolutions with a stride of 1, 64 filters and symmetric padding of 1. The final layer is a 1× 1 convolution. Our LSTM (Hochreiter & Schmidhuber, 1997) implementation is based on the variant used by Zaremba et al. (Zaremba et al., 2014). The input to the LSTM at each time step is the location of the agent in the, we separately embed the (x, y)-coordinates. This is then used to predict an additional channel in to the base CNN architecture described above. We also experimented with conditioning the initial hidden state on image features from a separate CNN, but found that this did not improve performance.
  In our demo conditional model, we preserve the spatial information of the demonstrations by feeding in the state visitation map as a image-grid, upsampled with bi-linear interpolation, as an additional channel to the image. In our setup, both the demo-conditional models share the same convolutional architecture, but differ only in how they encode condition on the demonstrations.
  For all our methods, we optimzed our model with Adam (Kingma & Ba, 2014). We turned over the learning rate α, the inner learning rate β of our approach and `2 weight decay on the initial parameters. In our LSTM learner, we experimented with different embedding sizes, as well as the dimensionality of the LSTM although we found that these hyperparameters did not impact performance. A negative result we found was that bias transformation (Finn et al., 2017b) did not help in our experimental setting.
sections.heading: B ENVIRONMENT DETAILS
sections.text: The sprites in our environment are extracted directly from the StarCraft files. We used in total 100 random units for meta-training. Evaluation on new objects was performed with 5 randomly selected sprites. For computational efficiency, we create a meta-training set of 1000 tasks and cache the optimal policy and state visitations under the true cost. Our evaluation is over 32 tasks. Our set of sprites was divided into two categories: buildings and characters. Each characters had multiple poses (taken from different frames of animation, such as walking/running/flying), whereas buildings only had a single pose. During meta-training the units were randomly placed, but to avoid the possibility
  that the agent would not need to actively avoid obstacles, the units were placed away from the boundary of the image in both the meta-validation and meta-test set.
  The terrain in each environment was randomly generated using a set of tiles, each belonging to a specific category (e.g. grass, dirt, water). For each tile, we also specified a set of possible tiles for each of the 4-neighbors. Using these constraints on the neighbors, we generated random environment terrains using a graph traversal algorithm, where successor tiles were sampled randomly from this set of possible tiles. This process resulted in randomly generated, seamless environments. The names of the units used in our experiments are as follows (names are from the original game files):
  The list of buildings used is: academy, assim, barrack, beacon, cerebrat, chemlab, chrysal, cocoon, comsat, control, depot, drydock, egg, extract, factory, fcolony, forge, gateway, genelab, geyser, hatchery, hive, infest, lair, larva, mutapit, nest, nexus, nukesilo, nydustpit, overlord, physics, probe, pylon, prism, pillbox, queen, rcluster, refinery, research, robotic, sbattery, scolony, spire, starbase, stargate, starport, temple, warm, weaponpl, wessel.
  The list of characters used is: acritter, arbiter, archives, archon, avenger, battlecr, brood, bugguy, carrier, civilian, defiler, dragoon, drone, dropship, firebat, gencore, ghost, guardian, hydra, intercep, jcritter, lurker, marine, missile, mutacham, mutalid, sapper, scout, scv, shuttle, snakey, spider, stank, tank, templar, trilob, ucereb, uikerr, ultra, vulture, witness, zealot, zergling.
sections.heading: C META-TEST TRAINING PERFORMANCE
sections.heading: D DETAILED META-OBJECTIVE DERIVATION
sections.text: We define the quality of reward function rθ parameterized by θ ∈ Rk on task T with the MaxEnt IRL loss, LTIRL(θ), described in Section 4. The corresponding gradient is
  ∇θLIRL(θ) = ∂ rθ ∂ θ (Eτ [µτ ]− µDT ), (9)
  where ∂ rθ/∂ θ is the k×|S||A|-dimensional Jacobian matrix of the reward function rθ with respect to the parameters θ. Here, µτ ∈ R|S||A| is the vector of state visitations under the trajectory τ (i.e. the vector whose elements are 1 if the corresponding state-action pair has been visited by the trajectory τ , and 0 otherwise), and µDT = 1 |DT | ∑ τ∈DT µτ is the mean state visitations over all demonstrated trajectories in DT . Let φT ∈ Rk be the updated parameters after a single gradient step. Then
  φT = θ − α∇θL trT (θ). (10)
  Let L testT be the MaxEnt IRL loss, where the expectation over trajectories is computed with respect to a test set that is disjoint from the set of demonstrations used to compute L testT (θ) in Eq. 10. We seek to minimize ∑
  T ∈T tr L testT (φT ) (11)
  over the parameters θ. To do so, we first compute the gradient of Eq. 11, which we derive here. Applying the chain rule
  ∇θL testT = ∂ φT ∂ θ ∂ rφT ∂ φT ∂ L testT ∂ rφT
  = ∂
  ∂ θ
  ( θ − α∇θL trT (θ) ) ∂ rφT ∂ φT ∂ L testT ∂ rφT
  = ( I− α ∂
  ∂ θ ( ∂ rθ ∂ θ (Eτ [µτ ]− µDT ) )) ∂ rφT ∂ φT ∂ L testT ∂ rφT
  (12)
  where in the last line we substitute in the gradient of the MaxEnt IRL loss in Eq. 9. In Eq. 12, we use the following notation:
  • ∂ φT /∂ θ denotes the k × k-dimensional vector of partial derivatives ∂ φT ,i/∂ θj ,
  • ∂ rφT /∂ φT denotes the k × |S||A|-dimensional matrix of partial derivatives ∂ rφT ,i/∂ φT ,j ,
  • and, ∂ L testT /∂ rφT denotes the k-dimensional gradient vector of L test T with respect to rφT .
  We will now focus on the term inside of the parentheses in Eq. 12, which is a k × k-dimensional matrix of partial derivatives.
  ∂
  ∂ θ ( ∂ rθ ∂ θ (Eτ [µτ ]− µDT ) ) = |S||A|∑ i=1 ∂2 rθ ∂ θ2 (Eτ [µτ ]− µDT )i + ( ∂ ∂ θ Eτ [µτ ] )( ∂ rθ ∂ θ )>
  = |S||A|∑ i=1 ∂2 rθ ∂ θ2 (Eτ [µτ ]− µDT )i + ( ∂ rθ ∂ θ )( ∂ ∂ rθ Eτ [µτ ] )( ∂ rθ ∂ θ )>
  where between the first and second lines, we apply the chain rule to expand the second term. In this expression, we make use of the following notation:
  • ∂2 rθ/∂ θ2 denotes the k × |S||A|-dimensional matrix of second-order partial derivatives of the form ∂2 rθ,i/∂ θ2j ,
  • (Eτ [µτ ] − µDT )i denotes the ith element of the |S||A|-dimensional vector (Eτ [µτ ] − µDT )i,
  • ∂ rθ/∂ θ denotes the k × |S||A|-dimensional matrix of partial derivatives of the form ∂ rθ,i/∂ θj ,
  • and, ∂∂ rθ Eτ [µτ ] is the |S||A|-dimensional Jacobian matrix of Eτ [µτ ] with respect to the reward function rθ (we will examine in more detail exactly what this is below).
  When substituted back into Eq. 12, the resulting gradient is equivalent to that in Eq. 8 in Section 4. In order to compute this gradient, however, we must take the gradient of the expectation Eτ [µτ ]
  with respect to the reward function rθ. This can be done by expanding the expectation as follows
  ∂
  ∂ rθ Eτ [µτ ] =
  ∂
  ∂ rθ ∑ τ ( exp(µ>τ rθ)∑ τ ′ exp(µ > τ ′rθ) ) µτ
  = ∑ τ (( exp(µ>τ rθ)∑ τ ′ exp(µ > τ ′rθ) ) (µτµ > τ )−
  exp(µ>τ rθ) ( ∑ τ ′ exp(µ > τ ′rθ)) 2 ∑ τ ′ (µτ ′µ > τ ) exp(µ > τ ′rθ) ) = ∑ τ P (τ | rθ)(µτµ>τ )− ∑ τ P (τ |rθ) ∑ τ ′ P (τ ′ | rθ)(µτ ′µ>τ )
  = Eτ [ (µτµ > τ )− ∑ τ ′ P (τ ′ | rθ)(µτ ′µ>τ ) ] = Eτ [µτµ>τ ]− Eτ ′,τ [µτ ′µ>τ ] = Eτ [µτµ>τ ]− Eτ [µτ ](Eτ [µτ ])>
  = Cov[µτ ].