summaries: The authors propose to learn a high level policy network that picks a subskill policy to predict actions with meta imitation learning (DMIL). Both the high and low level policy networks are fine-tuned at meta-test time. They show that DMIL converges and evaluate their approach against several ablations on the ML10 and ML45 setting of the meta-world benchmark. This paper proposes a meta imitation learning framework aimed at learning long-horizon robot control tasks with fast adaptation capabilities. Specifically, the proposed approach adapts the model-agnostic meta learning framework for learning a hierarchical policy, where both levels of the hierarchy are meta-trained and fine-tuned at test time to learn new tasks. On the metaworld benchmark, the proposed method outperforms prior work, and qualitative analysis shows that the method discovers meaningful skills. The paper describes Dual Meta Imitation Learning (DMIL) which is a hierarchical meta imitation learning method where the high-level network and sub-skills are iteratively meta-learned with model-agnostic meta-learning (MAML). The DMIL is a hierarchical extension of MAML-based imitation learning (IL) and is a meta-learning extension of Hierarchical Imitation learning (HIL).  The authors provide theoretical proof of the convergence based on the connection with the Expectation-Maximization (EM) algorithm. They showed the state-of-the-art few-shot imitation learning performance on the meta-world benchmark. The paper proposes an approach for few-shot imitation learning that jointly meta-learns a high-level policy and a set of low-level policies from a diverse set of demonstrations (multi-task). It then finetunes both high-level and low-level policies on few demonstrations of the target task for imitating them. In experiments on the MetaWorld50 task suite the proposed approach outperforms few-shot imitation methods that either don't use hierarchy or don't use meta-learning. 
abstractText: Hierarchical Imitation learning (HIL) is an effective way for robots to learn subskills from long-horizon unsegmented demonstrations. However, the learned hierarchical structure lacks the mechanism to transfer across multi-tasks or to new tasks, which makes them have to learn from scratch when facing a new situation. Transferring and reorganizing modular sub-skills require fast adaptation ability of the whole hierarchical structure. In this work, we propose Dual Meta Imitation Learning (DMIL), a hierarchical meta imitation learning method where the high-level network and sub-skills are iteratively meta-learned with model-agnostic meta-learning. DMIL uses the likelihood of state-action pairs from each sub-skill as the supervision for the high-level network adaptation, and use the adapted highlevel network to determine different data set for each sub-skill adaptation. We theoretically prove the convergence of the iterative training process of DMIL and establish the connection between DMIL and the Expectation-Maximization algorithm. Empirically, we achieve state-of-the-art few-shot imitation learning performance on the meta-world (Yu et al., 2019b) benchmark and comparable results on the Kitchen environment.
references.authors: Ferran Alet Tomás Lozano-Pérez Leslie Pack Kaelbling
references.title: Modular meta-learning
references.venue: In 2nd Annual Conference on Robot Learning,
references.authors: Christopher G. Atkeson Stefan Schaal
references.title: Robot learning from demonstration
references.venue: In Proceedings of the Fourteenth International Conference on Machine Learning,
references.authors: T Cachet J Perez S Kim
references.title: Transformer-based meta-imitation learning for robotic manipulation
references.venue: In Neural Information Processing Systems, Workshop on Robot Learning,
references.authors: Xi Chen Yan Duan Rein Houthooft John Schulman Ilya Sutskever Pieter Abbeel
references.title: Infogan: Interpretable representation learning by information maximizing generative adversarial nets
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Christian Daniel Herke van Hoof Jan Peters Gerhard Neumann
references.title: Probabilistic inference for determining options in reinforcement learning
references.venue: Machine Learning,
references.authors: Yan Duan Marcin Andrychowicz Bradly C. Stadie Jonathan Ho Jonas Schneider Ilya Sutskever Pieter Abbeel Wojciech Zaremba
references.title: One-shot imitation learning
references.venue: In Advances in Neural Information Processing Systems,
references.authors: David Duvenaud Dougal Maclaurin Ryan Adams
references.title: Early stopping as nonparametric variational inference
references.venue: In Artificial Intelligence and Statistics. PMLR,
references.authors: Alireza Fallah Aryan Mokhtari Asuman E. Ozdaglar
references.title: On the convergence theory of gradientbased model-agnostic meta-learning algorithms
references.venue: In The 23rd International Conference on Artificial Intelligence and Statistics,
references.authors: Chelsea Finn Pieter Abbeel Sergey Levine
references.title: Model-agnostic meta-learning for fast adaptation of deep networks
references.venue: In Proceedings of the 34th International Conference on Machine Learning,
references.authors: Chelsea Finn Tianhe Yu Tianhao Zhang Pieter Abbeel Sergey Levine
references.title: One-shot visual imitation learning via meta-learning
references.venue: In 1st Annual Conference on Robot Learning,
references.authors: Chelsea Finn Kelvin Xu Sergey Levine
references.title: Probabilistic model-agnostic meta-learning
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Carlos Florensa Yan Duan Pieter Abbeel
references.title: Stochastic neural networks for hierarchical reinforcement learning
references.venue: In 5th International Conference on Learning Representations,
references.authors: Kevin Frans Jonathan Ho Xi Chen Pieter Abbeel John Schulman
references.title: Meta learning shared hierarchies
references.venue: In 6th International Conference on Learning Representations,
references.authors: Justin Fu Aviral Kumar Ofir Nachum George Tucker Sergey Levine
references.title: D4rl: Datasets for deep data-driven reinforcement learning
references.venue: arXiv preprint arXiv:2004.07219,
references.authors: Seyed Kamyar Seyed Ghasemipour Shixiang Gu Richard S. Zemel
references.title: Smile: Scalable meta inverse reinforcement learning through context-conditional policies
references.venue: In Advances in Neural Information Processing Systems
references.authors: Erin Grant Chelsea Finn Sergey Levine Trevor Darrell Thomas L. Griffiths
references.title: Recasting gradient-based meta-learning as hierarchical bayes
references.venue: In 6th International Conference on Learning Representations,
references.authors: Abhishek Gupta Vikash Kumar Corey Lynch Sergey Levine Karol Hausman
references.title: Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning
references.authors: Tuomas Haarnoja Kristian Hartikainen Pieter Abbeel Sergey Levine
references.title: Latent space policies for hierarchical reinforcement learning
references.venue: Proceedings of the 35th International Conference on Machine Learning,
references.authors: Kourosh Hakhamaneshi Ruihan Zhao Albert Zhan Pieter Abbeel Michael Laskin
references.title: Hierarchical few-shot imitation with skill transition models
references.authors: Matthew J. Hausknecht Peter Stone
references.title: Deep reinforcement learning in parameterized action space
references.venue: In 4th International Conference on Learning Representations,
references.authors: Karol Hausman Jost Tobias Springenberg Ziyu Wang Nicolas Heess Martin A. Riedmiller
references.title: Learning an embedding space for transferable robot skills
references.venue: In 6th International Conference on Learning Representations,
references.authors: Peter Henderson Wei-Di Chang Pierre-Luc Bacon David Meger Joelle Pineau Doina Precup
references.title: Optiongan: Learning joint reward-policy options using generative adversarial inverse reinforcement learning
references.venue: In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence,
references.authors: Jonathan Ho Stefano Ermon
references.title: Generative adversarial imitation learning
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Robert A. Jacobs Michael I. Jordan Steven J. Nowlan Geoffrey E. Hinton
references.title: Adaptive mixtures of local experts
references.venue: Neural Computing,
references.authors: Mingxuan Jing Wenbing Huang Fuchun Sun Xiaojian Ma Tao Kong Chuang Gan Lei Li
references.title: Adversarial option-aware hierarchical imitation learning
references.venue: In Proceedings of the 38th International Conference on Machine Learning,
references.authors: Ilya Kostrikov Kumar Krishna Agrawal Sergey Levine Jonathan Tompson
references.title: Addressing sample inefficiency and reward bias in inverse reinforcement learning
references.venue: arXiv preprint arXiv:1809.02925,
references.authors: Sanjay Krishnan Roy Fox Ion Stoica Ken Goldberg
references.title: DDCO: discovery of deep continuous options for robot learning from demonstrations
references.venue: In 1st Annual Conference on Robot Learning,
references.authors: Hoang Minh Le Nan Jiang Alekh Agarwal Miroslav Dudı́k Yisong Yue Hal Daumé III
references.title: Hierarchical imitation and reinforcement learning
references.venue: In Proceedings of the 35th International Conference on Machine Learning,
references.authors: Sang-Hyun Lee Seung-Woo Seo
references.title: Learning compound tasks without task-specific knowledge via imitation and self-supervised learning
references.venue: In Proceedings of the 37th International Conference on Machine Learning,
references.authors: Yunzhu Li Jiaming Song Stefano Ermon
references.title: Infogail: Interpretable imitation learning from visual demonstrations
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Libin Liu Jessica K. Hodgins
references.title: Learning to schedule control fragments for physics-based characters using deep q-learning
references.venue: ACM Trans. Graph.,
references.authors: Corey Lynch Mohi Khansari Ted Xiao Vikash Kumar Jonathan Tompson Sergey Levine Pierre Sermanet
references.title: Learning latent plans from play
references.venue: In 3rd Annual Conference on Robot Learning,
references.authors: Josh Merel Arun Ahuja Vu Pham Saran Tunyasuvunakool Siqi Liu Dhruva Tirumala Nicolas Heess Greg Wayne
references.title: Hierarchical visuomotor control of humanoids
references.venue: In 7th International Conference on Learning Representations,
references.authors: Rafael Müller Simon Kornblith Geoffrey Hinton
references.title: When does label smoothing help
references.venue: arXiv preprint arXiv:1906.02629,
references.authors: Gerhard Neumann Wolfgang Maass Jan Peters
references.title: Learning complex motions by sequencing simpler motion templates
references.venue: Proceedings of the 26th Annual International Conference on Machine Learning,
references.authors: Takayuki Osa Joni Pajarinen Gerhard Neumann J. Andrew Bagnell Pieter Abbeel Jan Peters
references.title: An algorithmic perspective on imitation
references.authors: Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury Gregory Chanan Trevor Killeen Zeming Lin Natalia Gimelshein Luca Antiga
references.title: Pytorch: An imperative style, highperformance deep learning library
references.venue: Advances in neural information processing systems,
references.authors: Xue Bin Peng Pieter Abbeel Sergey Levine Michiel van de Panne
references.title: Deepmimic: exampleguided deep reinforcement learning of physics-based character skills
references.venue: ACM Trans. Graph.,
references.authors: Xue Bin Peng Michael Chang Grace Zhang Pieter Abbeel Sergey Levine
references.title: MCP: learning composable hierarchical control with multiplicative compositional policies
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Bob Price Craig Boutilier
references.title: Accelerating reinforcement learning through implicit imitation
references.venue: Journal of Artificial Intelligence Research,
references.authors: Sachin Ravi Alex Beatson
references.title: Amortized bayesian meta-learning
references.venue: In 7th International Conference on Learning Representations,
references.authors: Stéphane Ross Geoffrey J. Gordon Drew Bagnell
references.title: A reduction of imitation learning and structured prediction to no-regret online learning
references.venue: In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics,
references.authors: Reginaldo J Santos
references.title: Equivalence of regularization and truncated iteration for general ill-posed problems
references.venue: Linear algebra and its applications,
references.authors: Mohit Sharma Arjun Sharma Nicholas Rhinehart Kris M. Kitani
references.title: Directed-info GAIL: learning hierarchical policies from unsegmented demonstrations using directed information
references.venue: In 7th International Conference on Learning Representations,
references.authors: Tianmin Shu Caiming Xiong Richard Socher
references.title: Hierarchical and interpretable skill acquisition in multi-task reinforcement learning
references.venue: In 6th International Conference on Learning Representations,
references.authors: Richard S. Sutton Doina Precup Satinder P. Singh
references.title: Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning
references.venue: Artif. Intell.,
references.authors: Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser Illia Polosukhin
references.title: Attention is all you need
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Risto Vuorio Shao-Hua Sun Hexiang Hu Joseph J. Lim
references.title: Multimodal model-agnostic metalearning via task-aware modulation
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Kelvin Xu Ellis Ratner Anca D. Dragan Sergey Levine Chelsea Finn
references.title: Learning a prior over intent via meta-inverse reinforcement learning
references.venue: In Proceedings of the 36th International Conference on Machine Learning,
references.authors: Huaxiu Yao Ying Wei Junzhou Huang Zhenhui Li
references.title: Hierarchically structured meta-learning
references.venue: In Proceedings of the 36th International Conference on Machine Learning,
references.authors: Lantao Yu Tianhe Yu Chelsea Finn Stefano Ermon
references.title: Meta-inverse reinforcement learning with probabilistic context variables
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Tianhe Yu Pieter Abbeel Sergey Levine Chelsea Finn
references.title: One-shot hierarchical imitation learning of compound visuomotor tasks
references.venue: arXiv preprint arXiv:1810.11043,
references.authors: Tianhe Yu Chelsea Finn Sudeep Dasari Annie Xie Tianhao Zhang Pieter Abbeel Sergey Levine
references.title: One-shot imitation from observing humans via domain-adaptive meta-learning
references.venue: In Robotics: Science and Systems XIV,
references.authors: Tianhe Yu Deirdre Quillen Zhanpeng He Ryan Julian Karol Hausman Chelsea Finn Sergey Levine
references.title: Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning
references.venue: In 3rd Annual Conference on Robot Learning,
references.authors: Yuke Zhu Ziyu Wang Josh Merel Andrei A. Rusu Tom Erez Serkan Cabi Saran Tunyasuvunakool János Kramár Raia Hadsell Nando de Freitas Nicolas Heess
references.title: Reinforcement and imitation learning for diverse visuomotor skills
references.venue: In Robotics: Science and Systems XIV,
references.authors: Yayi Zou Xiaoqi Lu
references.title: Gradient-em bayesian meta-learning
references.venue: In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems,
references.authors: Grant
references.title: 2018), firstly in the case of linear models, early stopping of an iterative gradient descent process of λ equals to the maximum posterior estimation (MAP) (Santos
references.authors: Kostrikov
references.title: Hyper-parameter settings are available in table
references.venue: Ghasemipour et al
sections.text: Hierarchical Imitation learning (HIL) is an effective way for robots to learn subskills from long-horizon unsegmented demonstrations. However, the learned hierarchical structure lacks the mechanism to transfer across multi-tasks or to new tasks, which makes them have to learn from scratch when facing a new situation. Transferring and reorganizing modular sub-skills require fast adaptation ability of the whole hierarchical structure. In this work, we propose Dual Meta Imitation Learning (DMIL), a hierarchical meta imitation learning method where the high-level network and sub-skills are iteratively meta-learned with model-agnostic meta-learning. DMIL uses the likelihood of state-action pairs from each sub-skill as the supervision for the high-level network adaptation, and use the adapted highlevel network to determine different data set for each sub-skill adaptation. We theoretically prove the convergence of the iterative training process of DMIL and establish the connection between DMIL and the Expectation-Maximization algorithm. Empirically, we achieve state-of-the-art few-shot imitation learning performance on the meta-world (Yu et al., 2019b) benchmark and comparable results on the Kitchen environment.
sections.heading: 1 INTRODUCTION
sections.text: Imitation learning (IL) has shown promising results for intelligent robots to conveniently acquire skills from expert demonstrations (Zhu et al., 2018; Peng et al., 2018). Nevertheless, acquiring skills from long-horizon unsegmented demonstrations has been a challenge for IL algorithms, because of the well-known issue of compounding errors (Ross et al., 2011). This is one of the crucial problems for the application of IL methods to robots since plenty of manipulation tasks are long-horizon. Hierarchical Imitation Learning (HIL) aims to tackle this problem by decomposing long-horizon tasks with a hierarchical model, in which a set of sub-skills are employed to accomplish specific parts of the long-horizon task, and a high-level network is responsible for determining the switching of sub-skills along with the task. Such a hierarchical structure is usually modeled with Options (Daniel et al., 2016; Krishnan et al., 2017; Jing et al., 2021) or goal-conditioned structure (Le et al., 2018). HIL expresses the nature of how humans solve complex tasks, and has been considered to be a valuable direction for IL algorithms (Osa et al., 2018).
  However, most current HIL methods have no explicit mechanism to transfer previously learned skills to new tasks with few-shot demonstrations. This requirement comes from that the learned hierarchical structure may conflict with discrepant situations in new tasks. As shown in figure 1(a), both the high-level network and sub-skills need to be transferred to new forms to satisfy new requirements: the high-level network needs new manners to schedule sub-skills in new tasks (for example, calling different sub-skills at the same state), and each sub-skill needs to adapt to new specific forms in new tasks (for example, grasping different kinds of objects). Quickly adapting to new tasks can significantly increase the generalization ability of HIL methods and make them be applied to a wider range of scenarios, thus an appropriate approach needs to be exploited to endow HIL with such kind of ability with few-shot new task demonstrations.
  Recently, meta imitation learning (MIL) (Yu et al., 2018b; Finn et al., 2017b; Yu et al., 2018a) employ model-agnostic meta-learning (MAML) (Finn et al., 2017a) to imitation learning procedure to enable the learned policy to quickly adapt to new tasks with few shot demonstrations. MAML first fine-tunes the policy network in the inner loop, then evaluates the fine-tuned network to update its initial parameters with end-to-end gradient descent at the outer loop. This inspires us to integrate MAML into HIL to transfer the hierarchical structure. However, most MIL methods meta-learn a monolithic policy in an end-to-end fashion, which does not conform to the iterative training of the bi-level structure in HIL, where the training supervision of each level comes from the other level. Integrating MAML into such a self-supervised bi-level optimization procedure is a novel problem. The main challenge comes from how to schedule the inner loops and outer loops of the meta-learning process of bi-level modules in HIL to ensure convergence.
  In this work, we propose a novel hierarchical meta imitation learning framework called dual meta imitation learning (DMIL) to successfully incorporate MAML into the iterative training process of HIL, as shown in figure 1(b). We make the high-level network and sub-skills as mutual supervision during the bi-level MAML process: the likelihood of each state-action pair in few-shot demonstrations of each sub-skill can provide the supervisions for the meta-training of high-level network, and the fine-tuned high-level network in turn determines the data sets for the meta-training of each subskill, like a general EM process. We further design an elaborate training procedure for DMIL that first fine-tunes the high-level network and sub-skills in sequence at inner loops, then meta-updates them simultaneously at outer loops. We theoretically prove the convergence of this special training procedure by leveraging previous results from Ravi & Beatson (2019); Finn et al. (2018); Zou & Lu (2020) to reframe both MAML and DMIL as hierarchical Bayes inference processes and get the convergence of DMIL according to the convergence of MAML Fallah et al. (2020).
  We test our method on the challenging meta-world benchmark environments (Yu et al., 2019b) and the Kitchen environment of D4RL benchmarks (Fu et al., 2020). In our experiments, we successfully acquire a set of meaningful sub-skills from a large scale of manipulation tasks, and achieve state-ofthe-art few-shot imitation learning abilities in the ML45 suite. In summary, the main contributions of this paper are as follows:
  • We propose DMIL, a novel hierarchical meta imitation learning framework that meta-learns both the high-level network and sub-skills from unsegmented multi-task demonstrations.
  • We propose a special EM-like training algorithm to schedule the meta-learning and hierarchical imitation learning processes in DMIL and theoretically guarantee its convergence.
  • We achieve state-of-the-art few-shot imitation learning performance on meta-world benchmark environments and comparable results in the Kitchen environment.
sections.heading: 2 RELATED WORK
sections.heading: 2.1 HIERARCHICAL IMITATION LEARNING
sections.text: Recovering inherent sub-skills contained in expert demonstrations and then reuse them with hierarchical structures has long been an interesting topic in the hierarchical imitation learning (HIL) domain. According to whether there are pretraining tasks, we can divide HIL methods into two categories. The first kind aims to manually design a set of simple pretraining tasks that could encourage distinct skills or primitives, and then they learn a high-level network to master the switching of primitives to accomplish complex tasks (Florensa et al., 2017; Peng et al., 2019; Liu & Hodgins, 2017; Merel et al., 2019; Le et al., 2018). However, for unsegmented demonstrations where no pretraining tasks are provided, which is the situation in our paper, these methods can not be applied.
  The second kind of methods aim to learn sub-skills with unsupervised learning methods. Daniel et al. (2016); Krishnan et al. (2017) acquire Options (Sutton et al., 1999) from demonstrations with an Expectation-Maximization-like procedure and use the Baum-Welch algorithm to estimate the parameters of different options. Henderson et al. (2018); Jing et al. (2021) integrate generative adversarial networks into option discovery process. Li et al. (2017); Sharma et al. (2019); Lee & Seo (2020) incorporate generative-adversarial imitation learning (Ho & Ermon, 2016) framework and an information-theoretic metric (Chen et al., 2016) to simultaneously imitate the expert and maximize the mutual-information between latent sub-skill categories and corresponding trajectories to acquire decoupled sub-skills. There are some methods called mixture-of-expert (MoE) that compute the weighted sum of all primitives to get the action rather than only using one of them at each time step (Hausknecht & Stone, 2016; Neumann et al., 2009; Jacobs et al., 1991). Other methods aim to seek an appropriate latent space that can map sub-skills into it and then condition a policy on the latent variable to reuse sub-skills (Lynch et al., 2019; Haarnoja et al., 2018; Hausman et al., 2018; Hakhamaneshi et al., 2021; Price & Boutilier, 2003).
  For transferring learned sub-skills, some work fine-tune the whole structure in new tasks (Hakhamaneshi et al., 2021). However, the performance of fine-tuning all depends on the generalization of deep networks, which may vary among different tasks and network designs.
sections.heading: 2.2 META IMITATION LEARNING
sections.text: Meta imitation learning, or one-shot imitation learning, leverages various meta-learning methods and multi-task demonstrations to meta-learn a policy that can be quickly adapted to a new task with few-shot new task demonstrations. Duan et al. (2017); Cachet et al. (2020) employ self-attention modules to process the whole demonstration and the current observation to predict current action. Yu et al. (2018a); Finn et al. (2017b); Yu et al. (2018b) use model-agnostic meta-learning (MAML) (Finn et al., 2017a) to achieve one-shot imitation learning ability for various manipulation tasks with robot or human visual demonstrations. Xu et al. (2019); Yu et al. (2019a) propose to meta-learn a robust reward function that can be quickly adapted to new tasks and then use it to perform IRL in new tasks. However, they need downstream inverse reinforcement learning after the adaptation of reward functions, thus conflicts with our goal of few-shot adaptation. Most of above methods only learn one monolithic policy, lacking the ability to model multiple sub-skills in long-horizon tasks. Some works aim to tackle the multi-modal data problem in meta-learning by avoiding single parameters initialization across all tasks (Vuorio et al., 2019; Alet et al., 2018; Frans et al., 2018; Yao et al., 2019), but they lack the mechanism to schedule the switching of different sub-skills over time. There are some works that also meta-learn a set of sub-skills in a hierarchical structure (Yu et al., 2018a; Frans et al., 2018), but they either use manually designed pretraining tasks or relearn the high-level network in new tasks, which is not appropriate in few-shot imitation learning settings.
sections.heading: 3 METHOD
sections.heading: 3.1 PRELIMINARIES
sections.text: We denote a discrete-time finite-horizon Markov decision process (MDP) as a tuple (S,A, T, P, r, ρ0), where S is the state space, A is the action space, T is the time horizon,
  P : S × A × S → [0, 1] is the transition probability distribution, r : S × A → R is the reward function, and ρ0 is the distribution of the initial state s0.
sections.heading: 3.2 FORMULATION OF META IMITATION LEARNING PROBLEM
sections.text: We firstly introduce the general setting of the meta imitation learning problem. The goal of meta imitation learning is to extract some common knowledge from a set of robot manipulation tasks {Ti} that come from the same task distribution p(T ), and adapt it to new tasks quickly with few shot new task demonstrations. As in model-agnostic meta-learning algorithm (MAML) (Finn et al., 2017a), we formalize the common knowledge as the initial parameter θ of the policy network πθ that can be efficiently adapted with new task gradients.
  For each task Ti ∼ p(T ), a set of demonstrations Di is provided, where Di consists of N demonstration trajectories: Di = {τij}Nj=1, and τij consists of a sequence of state-action pairs: τij = {(st, at)} Tij t=1, where Tij is the length of τij . Each Di is randomly split into support set Dtri and query set Dvali for meta-training and meta-testing respectively. During the training phase, we sample m tasks from p(T ), and in each task Ti, we use Dtri to fine-tune πθ to get the adapted taskspecific parameters λi with gradient descent, and then evaluate it with Dvali to get the meta-gradient of Ti, and we optimize the initial parameters θ with the average of meta-gradients from all m tasks. As in Finn et al. (2017b); Duan et al. (2017), we use behavior cloning (Atkeson & Schaal, 1997) loss as our metrics for meta-training and meta-testing. It aims to train a policy πθ : S → A that maximizes the likelihood such that θ∗ = arg maxθ ∑N i=1 log πθ(ai|si), where N is the number of provided state-action pairs. We denote the loss function of this optimization problem as LBC(θ,D), and the general objective of meta imitation learning problem is:
  min θ m∑ i=1 LBC ( λi,Dvali ) , (1)
  where λi = θ − α∇θLBC(θ,Dtri ), and α is a hyper-parameter which represents the inner-update learning rate.
sections.heading: 3.3 DUAL META IMITATION LEARNING (DMIL)
sections.text: In this work we assume at each time step t, the robot may switch to different sub-skills to accomplish the task. We define the sub-skill category at each time step t as zt = 1, · · · ,K, where K is the maximum number of sub-skills. We assume a successful trajectory τij of a task Ti is generated from several (at least one) sub-skill policies, i.e., τij = ∑Tij t=1{(st, πE(st|zt))}, where πE represents the expert policy. Our goal is to learn a hierarchical structure from multi-task demonstrations {D1, · · · ,Dm} in an unsupervised fashion. In our model, a high-level network πθh that parameterized by θh determines the sub-skill category ẑt at each time step t, and the z-th sub-skill among K different sub-skills πθl1 , · · · , πθlK will be called to predict the corresponding action ât of state st, where the hat symbol denotes the predicted result. We use λh and λl1, · · · , λlk to represent the adapted parameters of θh and θl1, · · · , θlK respectively. In order to achieve few-shot learning ability in new tasks, we condition the high-level network only on states, i.e., ẑt = πθh(st), since at the testing phase we only have access to states and have no access to action information.
  DMIL aims to first fine-tune both πθh and πθl0 , · · · , πθlK and then meta-update them. In a new task, πθh may not provide correct sub-skill categories as stated in the introduction. However, sub-skills still retain the ability to give out supervision for the high-level network with knowledge learned from previously learned tasks and few-shot demonstrations. This is because most robot manipulation tasks are made up of a set of shared basis skills like reach, push and pick-place. As shown in the left side of 2, the sub-skill that gives out the closet ât to at can be seen as supervision for the high-level network to classify st into this sub-skill. On the other hand, the adapted high-level network can classify each data point in provided demonstrations to different sub-skills for them to perform finetuning, as shown in the right side of figure 2. In summary, DMIL contains four steps for one training iteration. We call them High-Inner-Update (HI), Low-Inner-Update (LI), High-Outer-Update (HO), and Low-Outer-Update (LO), which represents the fine-tuning and meta-updating process of the bi-level networks respectively. The key problem is how to arrange these optimization orders to ensure convergence. We first introduce these four steps formally here, then discuss how to schedule them in the next section. The whole procedure is summarized in algorithm 1.
  HI: For each sampled task Ti, we sample the first batch of trajectories {τi1} fromDtri . The principle of this step is to use sub-skill that can predict the closest action to the expert action to provide selfsupervised category ground truths for the training of high-level network, which is a classifier in form. We make every state-action pair passed directly to each sub-skill and compute LBC(θlk, τi1), k = 1, · · · ,K, and choose the ground truth at each time step as the sub-skill category k that minimizes LBC(θlk, (st, at)) :
  p(zi1t = k) = { 1, if k = arg mink LBC(θlk, (st, at)) 0, else . (2)
  Then we get predicted sub-skill categories from the high-level network: ˆzi1t = πθh(st), and use a cross-entropy loss to train the high-level network:
  Lh(θh, τi1) = − 1
  Ti1 Ti1∑ t=1 K∑ k=1 p(zi1t = k) log p( ˆzi1t = k). (3)
  Finally we perform gradient descent on the high-level network and get λh = θh−α∇θhLh(θh, τi1). Note θl1, · · · , θlk are freezed here. LI: We sample the second batch of trajectories {τi2} fromDtri . The adapted high level network πλh will process each state in τi2 to get sub-skill category ˆzi2t = πλh(st) at each time step, thus we get K separate data sets for different sub-skills: D2k = {(si2t, ai2t)| ˆzi2t = k}, k = 1, · · · ,K. Then we compute the adaptation loss for each sub-skill with the corresponding dataset. In case of continuous action space, we assume that actions belong to Gaussian distributions, so we have:
  LBC(θlk,D2k) = − 1
  Nk Nk∑ t=1 (at − πθlk(st))2, (4)
  where Nk is the number of state-action pairs in D2k. Finally we perform gradient descent on subskills and get λlk = θlk − α∇θlkLBC(θlk,D2k). Note πλh is frozen in this process. HO: We sample the third batch of trajectories {τi3} fromDtri and getL(λh, τi3) as in the HI process. Then we compute meta-gradient for θh with L(λh, τi3) as follows:
  ∇θhL(λh, τi3) = ∇λhL(λh, τi3)|λh=θh−α∇θhL(θh,τi2) ∗ ∇θhλh. (5)
  LO: we sample τi4 and get L(λlk,D4k), k = 1, · · · ,K as in the LI process, then we compute meta-gradient for θlk with L(λlk,D4k) as follows: ∇θlkL(λlk,D4k) = ∇λlkL(λlk,D4k)|λlk=θlk−α∇θlkL(θlk,D1k) ∗ ∇θlkλlk. (6) Note after the training of m tasks, we average all meta-gradients from m tasks and perform gradient descents on the initial parameters together to update high-level parameters θ′h = θh − β ∑m i=1∇θhL(λh, τi3) and sub-skill policies parameters θ′lk = θlk − β ∑m i=1∇θlkL(λlk, τi4), k = 1, · · · ,K, i.e., we do not update them at step 5 and 6. This is crucial to ensure convergence. For testing, although our method needs totally two batches of trajectories for one round of adaptation, in practice we find only using one trajectory to perform HI and LI also works well in new tasks, thus DMIL can satisfy the one-shot imitation learning requirement. Besides the above process, we also add an auxiliary loss to better drive out meaningful sub-skills to avoid the excessively frequently switching between different sub-skills along with time. Detailed information can be found in B.
sections.heading: 4 THEORETICAL ANALYSIS
sections.text: DMIL is a novel iterative hierarchical meta-learning procedure, and its convergence of DMIL needs to be proved to ensure feasibility. As stated in the above section, what makes DMIL special is that in HI and LI, we update parameters of each module immediately, but in LO and HO, we store the gradients of each part and update them simultaneously. In this section, we show this can make DMIL converge by rewriting both MAML and DMIL as hierarchical variational Bayes problems to establish the equivalence between them, since the convergence of MAML can be proved in Fallah et al. (2020). Proofs of all theorems are in Appendix C.
sections.heading: 4.1 HIERARCHICAL VARIATIONAL BAYES FORMULATION OF MAML
sections.text: According to (Ravi & Beatson, 2019), MAML is a hierarchical variational Bayes inference process. The general meta-learning objective 1 can be formulated as follows:
  L(θ, λ1, · · · , λm) = log [ m∏ i=1 p (Di|θ) ] ≥ m∑ i=1 {KL(q(φi;λi) ‖ p(φi|Di, θ))
  + Eq(φi;λi)[log p(Di, φi|θ)− log q(φi;λi)]}, (7)
  where φi, i = 1, · · · ,m represent the local latent variables for task Ti, and λ1, · · · , λM are the variational parameters of the approximate posteriors over φ1, · · · , φM . We denote λi as λi(Di, θ) and p(φi|Di, θ) as p(φi|Dtri , θ) to mean that λi and φi are determined with prior parameters θ and support data Dtri . First we need to minimize KL(q(φi;λi)‖p(φi|Dtri , θ)) w.r.t. λi. According to C.2, we have:
  λi(Dtri , θ) = arg max λi Eq(φi;λi)[log p(D tr i |φi)]−KL(q(φi;λi)‖p(φi|θ)), (8)
  and we can establish the connection between 8 and the fine-tuning process in MAML by the following Lemma:
  Lemma 1 In case q(φi;λi) is a Dirac-delta function and choosing Gaussian prior for p(φi|θ), equation 8 equals to the inner-update step of MAML, that is, maximizing log p(Dtri ) w.r.t. λi by earlystopping gradient-ascent with choosing µθ as initial point:
  λi(Dtri ; θ) = µθ + α∇θ log p ( Dtri |θ ) |θ=µθ . (9)
  Then we need to optimize L(θ, λ1, · · · , λM ) w.r.t. θ. Since we evaluate p(Di|λi(Dtri , θ)) with only Dvali , we assume p(Di|λi(Dtri , θ)) = p(Dvali |λi(Dtri , θ)). We give out the following theorem to establish the connection between the meta-update process and the optimization of L(θ, λ1, · · · , λM ): Theorem 1 In case that Σθ → 0+, i.e., the uncertainty in the global latent variables θ is small, the following equation holds:
  ∇θL(θ, λ1, · · · , λM ) = M∑ i=1 ∇λi log p(Dvali |λi) · ∇θλi(Dtri , θ). (10)
  A general EM algorithm will first compute the distribution of latent variables (E-step), then optimize the joint distribution of latent variable and trainable parameters (M-step), and the likelihood of data can be proved to be monotone increasing to guarantee the convergence since the evidence lower bound of likelihood is monotone increasing. Here φi, i = 1, · · · ,M are the latent variables, and θ corresponds to the trainable parameters. Lemma 1 and Theorem 1 correspond to the E-step and M-step respectively. In the following part we establish the equivalence between 9 with 3 and 4, and between 10 with 5 and 6 to prove the equivalence between DMIL and MAML.
sections.heading: 4.2 MODELING DMIL WITH HIERARCHICAL VARIATIONAL BAYES FRAMEWORK
sections.text: For simplicity, here we only derive in one specific task Ti, since derivatives of parameters from multi-task can directly add up. We first establish the connection between the maximization of log p(Dtri |θh, θl1, · · · , θlK) with the particular loss functions in DMIL:
  Theorem 2 In case of p(at|st, θlk) ∼ N (µθlk(st), σ2), we have:
  ∇θh log p(Dtri |θh, θl1, · · · , θlK) = ∇θhLh(θh,Dtri ), (11) and ∇θlk log p(Dtri |θh, θl1, · · · , θlK) = ∇θlkLBC(θlk,D2k), k = 1, · · · ,K. (12) Note in 12, D2k corresponds to data sets determined by the adapted high level network λh, and this connects with 3 and 4 in DMIL. According to 8, finding λi equals to maximize log p (Dtri |θ) in specific conditions, and here in Theorem 2, we prove that maximize log p(Dtri |θh, θl1, · · · , θlK) corresponds to 3 and 4 in DMIL. Thus theorem 2 corresponds to the E-step of DMIL, where we take τi1 and τi2 as Dtri , and optimize arg maxλi Eq(φi;λi)[log p(D tr i |φi)]−KL(q(φi;λi)‖p(φi|θ)) with coordinate descent method, which can be proved to be equal to 9 in C.5.
  For the M-step, we take τi3 and τi4 as Dvali . According to Theorem 1, we can take the derivative of λih, λil1, · · · , λilK to maximize the joint distribution of latent variables and trainable parameters to maximize the likelihood of dataset, so we have:
  ∇θh,θl log p(Dvali |λih, λil) = [∇λih log p(Dvali |λih) ∗ ∇θhλih(Dtri , θh), ∇λil log p(Dvali |λil) ∗ ∇θlλil(Dtri , θl)]T
  (13)
  where θil = [θi1, · · · , θiK ]T and λil = [λi1, · · · , λiK ]T . This is exactly the gradients computed in HO and LO steps. Note this computation process can be automatically accomplished with standard deep learning libraries such as PyTorch (Paszke et al., 2019). To this end, we establish the equivalence between DMIL and MAML, and the convergence of DMIL can be proved.
  For a clearer comparison, MAML is an iterative process of θ → λ → θ′, and DMIL is an iterative process of θh, θl → λh, θl → λh, λl → θ′h, θ′l, where the posterior estimation stages θh, θl → λh, θl → λh, λl has no effect on parameters θh, θl, thus can be divided to two steps as in DMIL. This decoupled fine-tuning fashion is exactly what we need to first adapt the high-level network and then adapt sub-skills. If we end-to-end fine-tune parameters like θh, θl → λh, λl, subskills will receive supervisions from an unadapted high-level network, which may provide incorrect classifications. Different to this, the meta-updating process λh, λl → θ′h, θ′l must be done at the same time, since if we update θh and θl successively, the later one will receive different derivative (for example, ∇θl log p(Dvali |θ′ih, λil) ) from derivatives in MAML (∇θl log p(Dvali |λih, λil)), and the equivalence would not be proved.
sections.heading: 5 EXPERIMENTS
sections.text: In experiments we aim to answer the following questions: (a) Can DMIL successfully transfer the learned hierarchical structure to new tasks with few-shot new task demonstrations? (b) Can DMIL achieve higher performance compared to other few-shot imitation learning methods? (c) What are effects of different parts in DMIL, such as the skill number K, the bi-level meta-learning procedure, and the continuity constraint? Video results are provided in supplementary materials.
sections.heading: 5.1 ENVIRONMENTS AND EXPERIMENT SETUPS
sections.text: We choose to evaluate DMIL on two representative robot manipulation environments. The first one is Meta-world benchmark environments (Yu et al., 2019b), which contains 50 diverse robot manipulation tasks, as shown in figure 6 and 7. We use both ML10 suite (10 meta-training tasks and 5 meta-testing tasks) and ML45 suite (45 meta-training tasks and 5 meta-testing tasks) to evaluate our method, and collect 2K demonstrations for each task. We choose Meta-world since we think a large scale of diverse manipulation tasks can help our method to learn semantic skills. We use the following approaches for comparison in this environment: Option-GAIL: a hierarchical generative adversarial imitation learning method to discover options from unsegmented demonstrations (Jing et al., 2021). We use Option-GAIL to evaluate the effect of meta-learning in DMIL. MIL: a transformer-based meta imitation learning method (Cachet et al., 2020). We use MIL to evaluate the effect of the hierarchical structure in DMIL. MLSH: the meta-learning shared hierarchies method (Frans et al., 2018) that relearns the high-level network in every new task. We use MLSH to evaluate the effect of fine-tune (rather than relearn) the high-level network in new tasks. PEMIRL: a contextual meta inverse RL method which transfers the reward function in the new tasks (Yu et al., 2019a). We use PEMIRL to show DMIL can transfer to new tasks that have significantly different reward functions.
  The second one is the Kitchen environment of the D4RL benchmark (Fu et al., 2020), which contains five different sub-tasks in the same kitchen environment. The accomplishment of each episode requires sequentially completions of four specific sub-tasks, as shown in figure 9. We use an open demonstration data set (Gupta et al., 2019) to train our method. During training, we exclude interactions with selected objects and at test-time we provide demonstrations that involve manipulating the excluded object to make them as unseen tasks. We choose this environment to show DMIL can be used in long-horizon tasks and is robust across different environments. We use two approaches for comparison in this experiment: SPiRL: an extension of the skill extraction methods to imitation learning over skill space (Price & Boutilier, 2003); FIST: an algorithm that extracts skills from offline data with an inverse dynamics model and a distance function (Hakhamaneshi et al., 2021).
  We use fully-connected neuron networks for both the high-level network and sub-skills. Detailed descriptions on the environment setup, demonstration collection procedure, hyper-parameters setting, training process, and descriptions of different methods can be found in appendix E.
sections.heading: 5.2 RESULTS
sections.text: Table 1 shows success rates of different methods in ML10 and ML45 suites with sub-skill number K = 3. We perform 1-shot and 3-shot experiments respectively to show the progressive few-shot performance of different methods. DMIL achieves the best results in ML10 testing suite and ML45 training and testing suites. This shows the superiority of our method for transferring across a large scale of manipulation tasks. OptionGAIL achieves high success rates in both ML10 and ML45 training suites. These results come from their hierarchical structures that have adequate capacity to fit potential multi-modal behaviors in multi-task demonstrations. MIL achieves comparable results for all meta-testing tasks but is worse than DMIL. This shows the necessity of meta-learning processes. Compared to them, PEMIRL and MLSH are mediocre among all suites. This comes from that the reward functions across different tasks are difficult to transfer with few shot demonstrations, and the relearned high-level network of MLSH damages previously learned knowledge. We also illustrate the t-sne results of these methods in figure 4(a) to further analyze the methods in appendix D.2.
  Table 2 shows the rewards of different methods on four unseen tasks in the Kitchen environment. FIST-no-FT refers to a variant of FIST that does not use future conditioning, which makes the comparison fairer. DMIL achieves higher rewards on two out of four tasks and comparable results on the other two tasks, which exhibits the effectiveness of the bi-level meta-training procedure. The poor performance of DMIL on the first task may come from the choice of skill number K or from low-quality demonstrations. We perform ablation studies on K in the next section.
  Figure 3 shows curves of sub-skill probabilities along time of two tasks window-close and peginsert-side of Meta-world and the microwave-kettle-top burner-light task in Kitchen environment. We can see the activation of sub-skills shows a strong correlation to different stages of tasks. In first two tasks, πθl0 activates when the robot is closing to something, πθl1 activates when the robot is picking up something, and πθl2 activates when the robot is manipulating something. In the third task, πθl2 activates when the robot is manipulating the microwave, πθl0 activates when the robot is
  manipulating the kettle or the light switch, and πθl3 activates when the robot is manipulating the burner switch. This shows that DMIL has the ability to learn semantic sub-skills from unsegmented multi-task demonstrations.
  5.3 ABLATION STUDIES
  Table 3: Success rates of different sub-skill number in Meta-world environments.
  ML10 ML45
  Meta-training Meta-testing Meta-training Meta-testing K 1-shot 3-shot 1-shot 3-shot 1-shot 3-shot 1-shot 3-shot
  2 0.76 0.955 0.32 0.72 0.563 0.818 0.44 0.67 3 0.775 0.949 0.396 0.71 0.59 0.859 0.376 0.64 5 0.795 0.94 0.52 0.57 0.713 0.92 0.21 0.48 10 0.8 0.975 0.38 0.62 0.736 0.931 0.34 0.64 In this section we perform ablation studies on different components of DMIL to provide effects of different parts. Due to limited space, we put ablations on finetuning steps, bi-level meta-learning processes, continuity constraint and hard/soft EM choices in appendix D.
  Effect of different skill number K: Table 3 shows the effect of different sub-skill number K in Meta-world experiments. We can see that a larger K can lead to higher success rates on metatraining tasks, but a smaller K can lead to better results on meta-testing tasks. This tells us that an excessive number of sub-skills may result in over-fitting on training data, and a smaller K can play the role of regularization. In Kitchen experiments, we can see similar phenomenons in table 5. It is worth noting in both environments, we did not encounter collapse problems, i.e., every sub-skill gets well-trained even when K = 8 in kitchen environment or K = 10 in Meta-world environments. This may come from that, in the meta-training stage, adding more sub-skills can help the whole structure get lower loss, thus DMIL will use all of them for training. However, in our supplementary videos, we can see sub-skills trained with a large K (for instance, K = 10 in Metaworld environments) are not as semantic as sub-skills trained by a small K (for instance, K = 3 in Meta-world environments) during the execution of a task.
sections.heading: 6 DISCUSSION AND FUTURE WORKS
sections.text: In this work, we propose DMIL to meta-learn a hierarchical structure from unsegmented multi-task demonstrations to endow it with fast adaptation ability to transfer to new tasks. We theoretically proved its convergence by reframing MAML and DMIL as hierarchical Bayes inference processes to get their equivalence. Empirically, we successfully acquire transferable hierarchical structures in both Meta-world and Kitchen Environments.
  The limitations of DMIL comes from several aspects, and future works can seek meaningful extensions in these perspectives. Firstly, DMIL models all tasks as bi-level structures. However, in real-world situations, tasks may be multi-level structures. One can extend DMIL to multi-level hierarchical structures like done in recent works (Shu et al., 2018). Secondly, DMIL does not capture temporal information in demonstrations. Future state conditioning in Hakhamaneshi et al. (2021) seems an effective tool to improve few-shot imitation learning performance in long-horizon tasks such as in the Kitchen environments. Future works can employ temporal module such as transformer (Vaswani et al., 2017) as the high-level network of DMIL to improve its performance.
sections.heading: 7 REPRODUCIBILITY STATEMENT
sections.text: We provide video results in supplementary materials. We provide some basic codes to run few-shot testing in meta-world environments in an anonymous repository 1. The complete training code will be provided later. Details of experiments such as environments, hyper-parameter setting, model setup, training procedure and data preprocessing can be found at E.
sections.heading: A ALGORITHM
sections.text: Algorithm 1 Dual Meta Imitation Learning Require: task distribution p(T ), multi-task demonstrations {Di}, i = 1, · · · ,m, initial parameters of high-level network θh and sub-skill policies θl1, · · · , θlK , inner and outer learning rate α, β. while not done do
  Sample batch of tasks Ti ∼ p(T ) for all Ti do
  Sample {τi1}, {τi2}, {τi3}, {τi4} from {Di} Evaluate∇θhLh(θh, τi1) according to 3 and τi1 Compute adapted parameters of high-level network: λh = θh − α∇θhLh(θh, τi1) Evaluate∇θlkLBC(θlk,D2k) according to 4 and τi2, k = 1, · · · ,K Compute adapted parameters of sub-skills: λlk = θlk−α∇θlkLBC(θlk,D2k), k = 1, · · · ,K Evaluate ∇θhLTi(λh, τi3) and ∇θlkLTi(λlk,D4k), k = 1, · · · ,K
  end for Update θh ← θh − β∇θh ∑ Ti∼p(T ) LTi(λh, τi3)
  Update θlk ← θlk − β∇θlk ∑ Ti∼p(T ) LTi(λlk,D4k), k = 1, · · · ,K
  end while
sections.heading: B AUXILIARY LOSS
sections.text: We adopt an auxiliary loss for DMIL to better drive out meaningful sub-skills by punishing excessive switching of sub-skills along the trajectory. This comes from an intuitive idea: each sub-skill should be a temporal-extended macro-action, and the high-level policy only needs to switch to different skills few times along a task, as the same idea of macro-action in MLSH Frans et al. (2018). We denote:
  sign(x) = { 1, if x = True 0, ifx = False , (14)
  and the auxiliary loss is:
  Laux(τ) = T−1∑ t=0 sign(ẑt+1 6= ẑt) / len(τ). (15)
  Although this operation seems discrete, in practice we can use the operations in modern deep learning framework such as PyTorch (Paszke et al., 2019) to make it differentiable. We add this loss function to the Lh(θh, τi1) and ∇θhL(λh, τi3) with a coefficient λ = 1. We also perform ablation studies of Laux and results are in table 7.
sections.heading: C PROOFS
sections.heading: C.1 PROOF OF LEMMA 1
sections.text: Lemma 1 In case q(φi;λi) is a Dirac-delta function and choosing Gaussian prior for p(φi|θ), equation 8 equals to the inner-update step of MAML, that is, maximizing log p(Dtri ) w.r.t. λi by earlystopping gradient-ascent with choosing µθ as initial point:
  λi(Dtri ; θ) = µθ + α∇θ log p ( Dtri |θ ) |θ=µθ . (16)
  Proof: in case of the conditions of Lemma 1, we have:
  λi(Dtri ; θ) = arg max λi [log p(Dtri |µλi)− ‖µλi − µθ‖ 2 /2Σ2θ], (17)
  As stated in Grant et al. (2018), firstly in the case of linear models, early stopping of an iterative gradient descent process of λ equals to the maximum posterior estimation (MAP) (Santos, 1996).
  In our case the posterior distribution refers to q(φi|λi), and MAML is a Bayes process to find the MAP estimate as the point estimate of λi(Dtri ; θ). In the nonlinear case, this point estimate is not necessarily the global mode of the posterior, and we can refer to Duvenaud et al. (2016) for another implicit posterior distribution over φi and making the early stopping procedure of MAML acting as priors to get the similar result.
sections.heading: C.2 PROOF OF EQUATION 8
sections.text: Equation 8 can be written as:
  λi(Dtri , θ) = arg min λi KL(q(φi;λi)‖p(φi|Dtri , θ))
  = arg max λi
  Eq(φi;λi)[log p(φi|D tr i , θ)− log q(φi;λi)]
  = arg max λi
  Eq(φi;λi)[log p(D tr i |φi)]−KL(q(φi;λi)‖p(φi|θ)),
  (18)
  where in MAML we assume p(Dtri |φi) = p(Dtri |φi, θ), and use the joint distribution p(Dtri , φi|θ) to replace p(φi|Dtri , θ) since we assume that p(Dtri ) subjects to uniform distribution. Thus 8 can be proved.
sections.heading: C.3 PROOF OF THEOREM 1
sections.text: Theorem 1 In case that Σθ → 0+, i.e., the uncertainty in the global latent variables θ is small, the following equation holds:
  ∇θL(θ, λ1, · · · , λM ) = M∑ i=1 ∇λi log p(Dvali |λi) ∗ ∇θλi(Dtri , θ). (19)
sections.heading: Proof:
sections.text: ∇θL(θ, λ1, · · · , λM ) ≈ M∑ i=1 {∇θEq(φi;λi)[log p(Di, φi|θ)− log q(φi;λi)]}
  = M∑ i=1 ∇θ[log p(Dvali |λi(Dtri , θ))− log p(λi(Dtri , θ)|θ)]
  ≈ M∑ i=1 ∇θ log p(Dvali |λi(Dtri , θ))
  = M∑ i=1 ∇λi log p(Dvali |λi) ∗ ∇θλi(Dtri , θ)
  (20)
  where the first approximate equal holds because the VI approximation error is small enough, and the second approximate equal holds because that in case Σθ → 0+ and assuming λi be a neuron network, log p(λi(Dtri , θ)|θ) ≈ 0 holds almost everywhere, so ∇θ log p(λi(Dtri , θ)|θ) ≈ 0. Note the condition of theorem 1 is usually satisfied since we are using MAML, and the initial parameters θ are assumed to be deterministic.
  From another perspective, the right side of above equation is the widely used meta-gradient in MAML, and it is equal to 1m ∑m i=1 ( I − α∇2θLBC (θ,Dtri ) ) ∗
  ∇λiLBC ( θ − α∇θLBC (θ,Dtri ) ,Dvali ) , which is proved to be converged by Fallah et al. (2020).
sections.heading: C.4 PROOF OF THEOREM 2
sections.text: Theorem 2 In case of p(at|st, θlk) ∼ N (µθlk(st), σ2), we have:
  ∇θh log p(Dtri |θh, θl1, · · · , θlK) = ∇θhLh(θh,Dtri ), (21)
  and
  ∇θlk log p(Dtri |θh, θl1, · · · , θlK) = ∇θlkLBC(θlk,D2k), k = 1, · · · ,K. (22)
  Proof: since p(Dtri |θh, θl1, · · · , θlK) = ∏N t=1 p(at|st, θh, θl1, · · · , θlK)p(st|θh, θl1, · · · , θlK) and the second term is independent of θ, we consider the first conditional probability:
  p(at|st, θh, θl1, · · · , θlK) = K∑ k=1 p(zk|st, θh)p(at|st, θlk). (23)
  In the HI step, p(at|st, θlk) is fixed, thus 23 becomes a convex optimization problem:
  max θh K∑ k=1 p(zk|st, θh)p(at|st, θ (24)
  The solution of this problem is λ∗h which satisfies p(zk|st, at, λ∗h) = 1, k = arg maxk p(at|st, θlk). This means that πθh needs to predict the sub-skill category at time step t as k, in which case πθlk can maximize p(at|st, θlk). In case we choose πθh to be a classifier that employs a Softmax layer at the end, minimizing the cross entropy loss 3 equals to maximize 24, thus 21 can be proved.
  In the LI step, p(zk|st, λh) is fixed, and the data sets for optimizing θl1, · · · , θlK are also fixed as D2k = {(sijt, aijt)| ˆzi2t = k}Nkt=1. Thus we need to maximize each p(at|st, θlk) with D2k. In case of p(at|st, θlk) ∼ N (µθlk(st), σ2) ∝ exp[− (at−πθlk (st)) 2
  2σ2 ], we have maxθlk p(at|st, θlk) ⇔ minθlk(at−πθlk(st))2, which leads to the loss function 4, thus 22 can be proved, which finishes the prove of Theorem 2.
sections.heading: C.5 PROOF OF THE E-STEP OF DMIL
sections.text: According to Theorem 1, we aim to maximize 17 w.r.t λi from the initial point θi with coordinate gradient ascent. We here need to prove that in DMIL, we could also achieve the global maximum point of λi as in MAML. We first give out the following Lemma:
  Lemma 2 Let x be the solution found by coordinate gradient descent of f(x). Let xi, i = 1, · · · , n be the n coordinate directions used in the optimization process. If f(x) can be decomposed as:
  f(x) = g(x) + n∑ i=1 hi(x), (25)
  where g(x) is a differentiable convex function, and each hi(x) is a convex function of the coordinate direction xi, then x is the global minimum of f(x).
  Proof: Let y be another arbitrary point, we have:
  f(y)− f(x) = g(y) + h(y)− (g(x) + h(x))
  ≥ ∇xg(x)T (y − x) + n∑ i=1 hi(yi)− hi(xi)
  = n∑ i=1 (∇ig(x)(yi − xi) + hi(yi)− hi(xi))
  ≥ 0.
  (26)
  Now let’s consider our problem. Consider
  log p(Dtri |θih, θil) = log T∑ t=1 p(at|st, θih, θil)p(st)
  = log T∑ t=1 p(st) K∑ k=1 p(zk|st, at, θih)p(at|st, θil)
  ≥ T∑ t=1 [log p(st) + log K∑ k=1 p(zk|st, at, θih)p(at|st, θil)]
  ≥ T∑ t=1 [log p(st) + K∑ k=1 log p(zk|st, at, θih)p(at|st, θil)]
  = T∑ t=1 [log p(st) + K∑ k=1 log p(zk|st, at, θih) + K∑ k=1 log p(at|st, θil)].
  (27)
  In our case, two coordinate directions are θih and θil. Let’s consider the terms inside the brackets. According to Lemma 2, we can think log p(st) as g(x)(here it equals to constant),∑K k=1 log p(zk|st, at, θih) as h1(x) and ∑K k=1 log p(at|st, θil) as h2(x). Thus the optimum can be proved.
sections.heading: D ADDITIONAL ABLATION STUDIES
sections.heading: D.1 EFFECTS OF THE BI-LEVEL META-LEARNING PROCESS
sections.text: We use two variants of DMIL to see the effectiveness of bi-level meta-earning process. DMILHigh: a variant that only meta-learns the high-level network, and DMIL-Low: a variant that only meta-learns sub-skills. We use Option-GAIL as a comparison that does not meta-learn any level of the hierarchical structure.
  Table 4 shows the results of this ablation study. DMIL-High achieves close results with OptionGAIL in all meta-training suites and better results in all meta-testing suites, but worse than DMIL in all cases. This shows that meta-learning the high-level network can help the hierarchical structure adapt to new tasks, but only transferring the high-level network is not enough for accomplish all kinds of new tasks. DMIL-Low achieves poor results in all suites except in ML10 meta-training suites. This shows that transferring the high-level network is necessary when training on a large scale of tasks or testing in new tasks.
  For better understand the effect of bi-level meta-learning process, we perform ablation study for three DMIL variants in an manually-designed new task push-around-wall (figure 8). In this task, the robot needs to grasp a cube and circle it around the wall. This is a brand new skill that is not in the meta-world suite. The accomplishment of this new task requires quickly adapting abilities of both the high-level network and sub-skills. We sample two demonstrations and use the first one as few-shot data, and illustrate the sub-skill categories of the second demonstration given by the high-level network at figure 4(b). Before adaptation, all variants give out approximately random results. However, after one-shot adaptation, DMIL classifies almost every state into sub-skill 0 and
  sub-skill 2, which indicates these two sub-skills in DMIL have been adapted to the new task, and the high-level network has also been adapted with the supervision from adapted sub-skills. Compared to DMIL, DMIL-Low still can not give out reasonable results after adaptation, since its high-level network lacks the ability to quickly transfer to new tasks. Instead, DMIL-High gives out plausible results after adaptation. This shows the high-level network has adapted to the new task according to the supervision from adapted sub-skills, but no sub-skill can dominate for a long time period since all sub-skills lack the quickly adaptation ability.
sections.heading: D.2 T-SNE RESULTS OF DIFFERENT METHODS
sections.text: For comparison of different methods, we illustrate the t-sne results of states of each sub-skill in an ML45 meta-testing task hand-insert in figure 4(a). We use 3 demonstrations for adaptation, and draw the t-sne results on another 16 demonstrations. This task is a meta-testing task, so no method has ever been trained on this task before.
  MLSH shows almost random clustering results no matter before and after adaptation, since its highlevel network is relearned in new tasks. OptionGAIL clusters to three sub-skills after adaptation. Compared to them, DMIL clusters the data to only two sub-skills after adaptation. We believe fewer categories reflect more meaningful sub-skills are developed in DMIL.
sections.heading: D.3 EFFECTS OF SUB-SKILL NUMBER K IN THE KITCHEN ENVIRONMENT
sections.text: We perform ablation studies of sub-skill number K on the Kitchen environments and choose K = 2, 4, 8 respectively. Table 5 shows the results. We can see that a smaller number of sub-skills can achieve better results on such four unseen results that a large number of sub-skills. This may indicate that the sub-skill number K can work as a ’bottleneck’ like the middle layer in an auto-encoder.
sections.heading: D.4 EFFECTS OF FINE-TUNING STEPS
sections.text: As all few-shot learning problems, the fine-tuning steps in new tasks to some extent determine the performance of the trained model. It controls the balance between under-fitting and over-fitting. We perform ablation studies of fine-tune steps in Meta-world benchmarks with K = 5 and lr=1e-2 in table 6. Results are as follows:
sections.heading: D.5 EFFECTS OF CONTINUITY REGULARIZATION
sections.text: We perform ablation studies of the effect of continuity regularization as following table 7. DMIL nc means no continuity regularization. We can see that the continuity constraint would damage metatraining performance slightly, but increase the meta-testing performance greatly.
sections.heading: D.6 EFFECTS OF HARD/SOFT EM CHOICES
sections.text: In DMIL, we use hard EM algorithm to train the high-level network. One may think about to use soft cross entropy loss to train the high-level network to get better results. We perform this ablation study in the following table 8. We can see that a soft cross entropy training won’t help increase the whole success rates. This may comes from that, usually we use soft cross entropy (such as label smoothing) to prevent over-fitting. However, in our situation, this may cause under-fitting, since training on such a large scale of diverse manipulation tasks is already very difficult. Future works can seek more comparisons about this choice.
  Additionally, we found an interesting phenomena that the training loss of the high-level network with a softmax shows a trend of rising first and then falling, as shown in . In our experiments, a softmax loss may regularize the optimization process to make the high-level network be under-fitting on training data. This may come from that, the experiment environment (Meta-world) contains a large scale of manipulation tasks, in which the training of the high-level network can be difficult and unstable. Thus a soft-max cross entropy loss cannot help that much here like how it works as a regularizer to prevent over-fitting in the label smoothing (Müller et al., 2019).
sections.heading: E EXPERIMENT DETAILS
sections.heading: E.1 ENVIRONMENTS
sections.text: See figure 7,6,9 and 8.
  Figure 6: The ML45 environment.
sections.heading: E.2 MODEL SETUP
sections.text: DMIL: The high-level network and each sub-skill is modeled with a 4-layer fully-connected neuron network, with 512 ReLU units in each layer. We use Adam as the meta-optimizer. DMIL-High and
  DMIL-Low use the same architecture with DMIL. Hyper-parameter settings are available in table 9.
  MIL: We use a transformer (Vaswani et al., 2017) as the policy to perform MAML. The input of the encoder is the whole one-shot demonstration or 3-shot demonstrations with concatenated state and action. The input of the decoder is current state. The output is the predicted action. Hyper-parameter settings are available in table 10.
  MLSH: We use the same settings of network with DMIL here. The macro step of high-level network is 3. Since our problem is not a reinforcement learning process, we use the behavior cloning variant of MLSH. The pseudo reward is defined by the negative mean square loss of the predicted action and the ground truth, and we perform pseudo reinforcement learning process with off-policy demonstration data. We use PPO as our reinforcement learning algorithm. Note in this way we ignore the importance sampling weights that required by replacing the sampling process in the environments with the demonstrations in the replay buffer, which has been shown to be effective in practice in
  Kostrikov et al. (2018); Ghasemipour et al. (2019). Hyper-parameter settings are available in table 11.
  PEMIRL: We use the same setting of the high-level network as the policy πω and the inference model qψ in PEMIRL. We use PPO as our reinforcement learning algorithm. We use a 3-layer fullyconnected neuron network as the context-dependent disentangled reward estimator rθ(s,m) and the context-dependent potential function hφ(s,m). Here we also use the behavior cloning variant of PEMIRL to only train models on the off-policy data. Hyper-parameter settings are available in table 12.
sections.heading: E.3 TRAINING DETAILS
sections.text: For fine-tuning, OptionGAIL, DMIL-Low and DMIL-High have no meta-learning mechanism for (some parts of) the trained model. In the few-shot adaptation process, we have different fine-tune method for these baselines:
  OptionGAIL: We train OptionGAIL models on the provided few-shot demonstrations for a few epochs.
  DMIL-Low: We fix the high-level network and only fine-tune sub-skills with few-shot demonstrations.
  DMIL-High: We fix sub-skills and only fine-tune the high-level network with few-shot demonstrations.
sections.heading: E.4 DETAILED RESULTS
sections.text: We provided detailed success rates of each environments of different methods in figure 10 and 11 with K = 3.