We present a framework for learning hierarchical policies from demonstrations, using sparse natural language annotations to guide the discovery of reusable skills for autonomous decision-making. We formulate a generative model of action sequences in which goals generate sequences of high-level subtask descriptions, and these descriptions generate sequences of low-level actions. We describe how to train this model using primarily unannotated demonstrations by parsing demonstrations into sequences of named high-level subtasks, using only a small number of seed annotations to ground language in action. In trained models, natural language commands index a combinatorial library of skills; agents can use these skills to plan by generating high-level instruction sequences tailored to novel goals. We evaluate this approach in the ALFRED household simulation environment, providing natural language annotations for only 10% of demonstrations. It achieves task completion rates comparable to state-of-the-art models (outperforming several recent methods with access to ground-truth plans during training and evaluation) while providing structured and human-readable high-level plans. 1 1 Introduction Building autonomous agents that integrate high-level reasoning with low-level perception and control is a long-standing challenge in artificial intelligence (Fikes et al., 1972; Newell, 1973; Sacerdoti, 1973; Brockett, 1993). Fig. 1 shows an example: to accomplish a task such as cooking an egg , an agent must first find the egg , then grasp it , then locate a stove or microwave , at each step reasoning about both these subtasks and complex, unstructured sensor data. Hierarchical planning models (e.g. Sutton et al., 1999)which first reason about abstract 1 Code and visualizations: https://sites.google.com/ view/skill-induction-latent-lang/. states and actions, then ground these in concrete control decisionsplay a key role in most existing agent architectures. But training effective hierarchical models for general environments and goals remains difficult. Standard techniques either require detailed formal task specifications, limiting their applicability in complex and hard-to-formalize environments, or are restricted to extremely simple high-level actions, limiting their expressive power (Bacon et al., 2017; Sutton et al., 1999; Dietterich, 1999; Kaelbling and Lozano-Prez, 2011). Several recent papers have proposed to overcome these limitations using richer forms of supervision especially languageas a scaffold for hierarchical policy learning. In latent language policies (LLPs; Andreas et al., 2018), controllers first map 1713 from high-level goals to sequences of natural language instructions, then use instruction following models to translate those instructions into actions. But applications of language-based supervision for long-horizon policy learning have remained quite limited in scope. Current LLP training approaches treat language as a latent variable only during prediction, and require fully supervised (and often impractically large) datasets that align goal specifications with instructions and instructions with low-level actions. As a result, all existing work on language-based policy learning has focused on very short time horizons (Andreas et al., 2018), restricted language (Hu et al., 2019; Jacob et al., 2021) or synthetic training data (Shu et al., 2018; Jiang et al., 2019). In this paper, we show that it is possible to train language-based hierarchical policies that outperform state-of-the-art baselines using only minimal natural language supervision. We introduce a procedure for weakly and partially supervised training of LLPs using ungrounded text corpora, unlabeled demonstrations, and a small set of annotations linking the two. To do so, we model training demonstrations as generated by latent high-level plans: we describe a deep, structured latent variable model in which goals generate subtask descriptions and subtask descriptions generate actions. We show how to learn in this model by performing inference in the infinite, combinatorial space of latent plans while using a comparatively small set of annotated demonstrations to seed the learning process. Using an extremely reduced version of the ALFRED household robotics dataset (Shridhar et al., 2020)with 10% of labeled training instructions, no alignments during training, and no instructions at all during evaluationour approach performs comparably a state-of-the-art model that makes much stronger dataset-specific assumptions (Blukis et al., 2021), while outperforming several models (Zhang and Chai, 2021; Suglia et al., 2021; Kim et al., 2021) that use more information during both training and evaluation . Our method correctly segments and labels subtasks in unlabeled demonstrations, including subtasks that involve novel compositions of actions and objects. Additional experiments show that pretraining on large (ungrounded) text corpora (Raffel et al., 2020) contributes to this success, demonstrating one mechanism by which background knowledge encoded in language can benefit tasks that do not involve language as an input or an output. Indeed, our results show that relatively little information about language grounding is needed for effective learning of language-based policiesa rich model of natural language text, a large number of demonstrations, and a small number of annotations suffice for learning compositional libraries of skills and effective policies for deploying them. We consider learning problems in which agents must perform multi-step tasks (like cooking an egg ; Fig. 1) in interactive environments. We formalize these problems as undiscounted, episodic, partially observed Markov decision processes (POMDPs) defined by a tuple ( S , A , T,  , O ) , where S is a set of states , A is a set of actions , T : S  A  S is an (unknown) state transition function ,  is a set of observations , and O : S   is an (unknown) observation function . 2 We assume that observations include a distinguished goal specification g that remains constant throughout an episode; given a dataset D of consisting of goals g and demonstrations d (i.e. D = { ( d 1 , g 1 ) , ( d 2 , g 2 ) . . . } ; d = [( o 1 , a 1 ) , ( o 2 , a 2 ) , . . . ]; o   , a  A ), we aim to learn a goal-conditional policy  ( a t | a : t  1 , o : t , g ) =  ( a t | a 1 , . . . , a t  1 , o 1 , . . . , o t , g ) that generalizes demonstrated behaviors to novel goals and states. For tasks like the ones depicted in Fig. 1, this learning problem requires agents to accomplish multiple subgoals (like finding an egg or operating an appliance ) in a feasible sequence. As in past work, we address this challenge by focusing on hierarchical policy representations that plan over temporal abstractions of low-level action sequences. We consider a generic class of hierarchical policies that first predict a sequence of subtask specifications  from a distribution  C (  i |  : i  1 , g ) (the controller ), then from each  generate a sequence of actions a 1 . . . a n from a distribution  E ( a i | a : i  1 , o : i ,  ) (the executor ). 3 At each timestep,  E may either generate an action from A ; or a special termination signal STOP ; after STOP is selected, control is returned to  C and a new  is generated. This process is visualized 2 For notational convenience, we assume without loss of generality that T and O are deterministic. 3 In past work,  E often conditions on the current observation as well as goal and history of past subtask specifications; we found that this extra information was not needed for the tasks studied here. (a) Figure 2: (a) When a hierarchical policy is deployed,  C generates a sequence of subtask specifications, and  E translates each of these to a low-level action sequence ending in STOP . At training time, this hierarchical structure is not available, and must be inferred to train our model. To do so, we assign each action a i an auxiliary alignment variable  i identifying the subtask that produced it. Alignments divide an action sequence into a sequence of segments s containing actions aligned to the same subtask. Automatically segmenting training demonstrations makes it possible to learn modular, reusable policies for individual subtasks without direct supervision. (b) Overview of the proposed learning algorithm (SL) 3 , which alternates between segmenting (by aligning) actions to fixed subtask specifications; labeling segments given fixed alignments, and updating model parameters. in Fig. 2(a). Trajectories generated by hierarchical policies themselves have hierarchical structure: each subtask specification  generates a segment of a trajectory (delimited by a STOP action) that accomplishes a specific subgoal. Training a hierarchical policy requires first defin-ing a space of subtask specifications  , then parameterizing controller and executor policies that can generate these specifications appropriately. Most past research has either pre-defined an inventory of target skills and independently supervised  C and  E (Sutton et al., 1999; Kulkarni et al., 2016; Dayan and Hinton, 1992); or performed unsupervised discovery of a finite skill inventory using clustering techniques (Dietterich, 1999; Fox et al., 2017). Both methods have limitations, and recent work has explored methods for using richer supervision to guide discovery of skills that are more robust than human-specified ones and more generalizable than automatically discovered ones. One frequently proposed source of supervision is language: in latent language policies ,  C is trained to generate goal-relevant instructions in natural language,  E is trained to follow instructions, and the space of abstract actions available for planning is in principle as structured and expressive as language itself. But current approaches to LLP training remain impractical, requiring large datasets of independent, fine-grained supervision for  C and  E . Below, we describe how to overcome this limitation, and instead learn from large collections of unlabeled demonstrations augmented with only a small amount of natural language supervision. Overview We train hierarchical policies on unannotated action sequences by inferring latent natural language descriptions of the subtasks they accomplish (Fig. 2(b)). We present a learning algorithm that jointly partitions these action sequences into smaller segments exhibiting reusable, task-general skills, labels each segment with a description, trains  C to generate subtask descriptions from goals, and  E to generate actions from subtask descriptions. Formally, we assume access to two kinds of training data: a large collection of unannotated demonstrations D = { ( d 1 , g 1 ) , ( d 2 , g 2 ) , . . . } and a smaller collection of annotated demonstrations D ann = { ( d 1 , g 1 ,  1 ) , ( d 2 , g 2 ,  2 ) , . . . } where each  consists of a sequence of natural language instructions [  1 ,  2 , . . . ] corresponding to the subtask sequence that should be generated by  C . We assume that even annotated trajectories leave much of the structure depicted in Fig. 2(a) unspecified, containing no explicit segmentations or STOP markers. (The number of instructions |  | will in general be smaller than the number of actions | d | .) Training  E requires inferring the correspondence between actions and annotations on D ann while inferring annotations themselves on D . Training objective To begin, it will be convenient to have an explicit expression for the probability of a demonstration given a policy (  C ,  E ) . To do so, we first observe that the hierarchical generation procedure depicted in Fig. 2(a) produces a latent alignment between each action and the subtask 1715  that generated it. We denote these alignments  , writing  i = j to indicate that a i was generated from  j . Because  C executes subtasks in sequence, alignments are monotonic , satisfying  i =  i  1 or  i =  i  1 + 1 . Let seg(  ) denote the segmentation associated with  , the sequence of sequences of action indices [[ i :  i = 1] , [ i :  i = 2] , . . . ] aligned to the same instruction (see Fig. 2(a)). Then, for a fixed policy and POMDP, we may write the joint probability of a demonstration, goal, annotation, and alignment as: p ( d , g,  ,  )  (cid:89) s  seg(  ) (cid:20)  C (  s |  < s , g )  (cid:16) (cid:89) i  1 .. | s |  E ( a i | a s : i  1 , o s : i ,   i ) (cid:17)   E ( STOP | a s , o s ) (cid:21) . (1) Here < s (in a slight abuse of notation) denotes all segments preceding s , and s i is the index of the i th action in s . The constant of proportionality in Eq. (1) depends only on terms involving T ( s (cid:48) | s, a ) , O ( o | s ) and p ( g ) , all independent of  C or  E ; Eq. (1) thus describes the component of the data likelihood under the agent's control (Ziebart et al., 2013). With this definition, and given D and D ann as defined above, we may train a latent language policy using partial natural language annotations via ordinary maximum likelihood estimation, imputing the missing segmentations and labels in the training set jointly with the parameters of  C and  E (which we denote  ) in the combined annotated and unannotated likelihoods: arg max   ,   ,   L (  ,   ,   ) + L ann (  ,   ) (2) where L (  ,   ,   ) = (cid:88) ( d ,g ) D log p ( d , g,   ,   ) (3) L ann (  ,   ) = (cid:88) ( d ,g,  ) D ann log p ( d , g,  ,   ) (4) and where we have suppressed the dependence of p ( d , g,  ,  ) on   for clarity. This objective involves continuous parameters   , discrete alignments   , and discrete labelings   . We optimize it via block coordinate ascent on each of these components in turn: alternating between resegmenting demonstrations, relabeling those without ground-truth labels, and updating parameters . The full learning algorithm, which we refer to as (SL) 3 ( semi-supervised skill learning with latent language ), is shown in Algorithm 1, with each step of the optimization procedure described in more detail below. The segmentation step associates each low-level action with a high-level subtask by finding the highest scoring alignment sequence  for each demonstration in D and D ann . While the number of possible alignments for a single demonstration is exponential in demonstration length, the assumption that  E depends only on the current subtask implies the following recurrence relation: max  1: n p ( d 1: n , g,  1: m ,  1: n ) = max i (cid:16) max  1: i p ( d 1: i , g,  1: m  1 ,  1: i )  p ( d i +1: n , g,  m ,  i +1: n = m ) (cid:17) (5) This means that the highest-scoring segmentation can be computed by an algorithm that recursively identifies the highest-scoring alignment to each pre-fix of the instruction sequence at each action (Al-gorithm 2), a process requiring O ( | d ||  | ) space and O ( | d | 2 |  | ) time. The structure of this dynamic program is identical to the forward algorithm for hidden semi-Markov models (HSMMs), which are widely used in NLP for tasks like language generation and word alignment (Wiseman et al., 2018). Indeed, Algorithm 2 can be derived immediately from Eq. (1) by interpreting p ( d , g,  ,  ) as the output distribution for an HSMM in which emissions are actions, hidden states are alignments, the emission distribution is  E and the transition distribution is the deterministic distribution with p (  + 1 |  ) = 1 . This segmentation procedure does not produce meaningful subtask boundaries until an initial executor policy has been trained. Thus, during the first iteration of training, we estimate a segmentation by by fitting a 3-state hidden Markov model to training action sequences using the BaumWelch algorithm (Baum et al., 1970), and mark state transitions as segment boundaries. Details about the initialization step may be found in Appendix B. 1716 Algorithm 1: (SL) 3 : Semi-Supervised Skill Learning with Latent Language Input : Unannotated demonstrations D = { ( d 1 , g 1 ) , ( d 2 , g 2 ) , . . . } ; Annotated demonstrations D ann = { ( d 1 , g 1 ,  1 ) , ( d 2 , g 2 ,  2 ) , . . . } Output : Inferred alignments   , labels   , and parameters  for  C and  E . // Segmentation // Infer alignments between actions and subtasks. if t = 1 then Initialize   using the BaumWelch algorithm (Baum et al., 1970) else    arg max   L (  ,   ,   ) + L ann (  ,   ) [Algorithm 2]. end // Labeling // Infer subtask labels for unannotated demos D . Output : Maximum a posteriori alignments  . from scores via back-tracing (Rabiner, 1989). arg max   ( ,  ,  ) Inference of latent, language-based plan descriptions in unannotated demonstrations involves an intractable search over string-valued  . To approximate this search tractably, we used a learned, amortized inference procedure (Wainwright and Jordan, 2008; Hoffman et al., 2013; Kingma and Welling, 2014) to impute descriptions given fixed segmentations. During each parameter update step (described below), we train an inference model q  (  | a s ( i ) , a s ( i +1) , g ) to approximate the posterior distribution over descriptions for a given segment given a goal, the segment's actions, and the actions from the subsequent segment. 4 Then, during the labeling step, we label complete demonstrations by choosing the highest-scoring instruction for each trajectory independently: arg max  log p ( d , g,  ,  )  (cid:104) arg max  q (  | a s ( i ) , a s ( i +1) , g ) (cid:12)(cid:12)(cid:12) s ( i )  seg(  ) (cid:105) (6) Labeling is performed only for demonstrations in D , leaving the labels for D ann fixed during training. Param update: arg max   L (  ,   ,   )+ L ann (  ,   ) This is the simplest of the three update steps: given fixed instructions and alignments, and  E ,  C parameterized as neural networks, this objective is differentiable end-to-end. In each iteration, we train these to convergence (optimization details are described in Section 4 and Appendix C). During the parameter update step, we also fit parameters  of the proposal model to maximize the likelihood (cid:80) d (cid:80) s , log q  (  | a s , o s ) with respect to the current segmentations  s and labels   . As goals, subtask indicators, and actions may all be encoded as natural language strings,  C and  E may be implemented as conditional language models. As described below, we initialize both policies with models pretrained on a large text corpora. Our experiments aim to answer two questions. First, does the latent-language policy representation described in Section 3 improve downstream performance on complex tasks? Second, how many natural language annotations are needed to train 4 In our experiments, conditioning on observations or longer context did not improve the accuracy of this model. Environment We investigate these questions in the ALFRED environment of Shridhar et al. (2020). ALFRED consists of a set of interactive simulated households containing a total of 120 rooms, accompanied by a dataset of 8,055 expert task demonstrations for an embodied agent annotated with 25,743 English-language instructions. Observations o are bitmap images from a forward-facing camera, and actions a are drawn from a set of 12 low-level navigation and manipulation primitives. Manipulation actions (7 of the 12) additionally require predicting a mask over the visual input to select an object for interaction. See Shridhar et al. (2020) for details. While the ALFRED environment is typically used to evaluate instruction following models, which map from detailed, step-by-step natural language descriptions to action sequences (Shridhar et al., 2020; Singh et al., 2020; Corona et al., 2021), our experiments focus on an goal-only evaluation in which agents are given goals (but not fine-grained instructions) at test time. Several previous studies have also considered goal-only evaluation for ALFRED, but most use extremely fine-grained supervision at training time , including full supervision of symbolic plan representations and their alignments to demonstrations (Min et al., 2021; Zhang and Chai, 2021), or derived sub-task segmentations using ALFRED-specific rules (Blukis et al., 2021). In contrast, our approach supports learning from partial, language-based annotations without segmentations or alignments, and this data condition is the main focus of our evaluation. Modeling details  C and  E are implemented as sequence-to-sequence transformer networks (Vaswani et al., 2017). C , which maps from text-based goal specifications to text-based instruction sequences, is initialized with a pre-trained T5-small language model (Raffel et al., 2020). E , which maps from (textual) instructions and (image-based) observations to (textual) actions and (image-based) object selection masks is also initialized with T5-small ; to incorporate visual input, this model first embeds observations using a pretrained ResNet18 model (He et al., 2016) and transforms these linearly to the same dimensionality as the word embedding layer. Details about the architecture of  C and  E may be found in Appendix C. Model variants for exploration In ALFRED, navigation in the goal-only condition requires exploration of the environment, but no exploration is demonstrated in training data, and techniques other than imitation learning are required for this specific skill. To reflect this, we replace all annotations containing detailed navigation instructions go to the glass on the table to your left with generic ones find a glass . Examples and details of how navigation instructions are modified can be found in Appendix E and Fig. 7. The ordinary (SL) 3 model described above is trained on these abstracted instructions. A key advantage of (SL) 3 is modularity: individual skills may be independently supervised or reimplemented. To further improve (SL) 3 's navigation capabilities, we introduce two model variants in which sub-task specifications beginning Find. . . are executed by a either a planner with ground-truth environment information or a specialized navigation module from the HLSM model (Blukis et al., 2021) rather than  E . Outside of navigation, these models preserve the architecture and training procedure of (SL) 3 , and are labeled (SL) 3 +planner and (SL) 3 +HLSM in experiments below. seq2seq : A standard (non-hierarchical) goal-conditioned policy, trained on the ( g, d ) pairs in D  D ann to maximize (cid:80) a , o ,g log  ( a | o , g ) , with  parameterized similar to  E . seq2seq2seq : A supervised hierarchical policy with the same architectures for  C and  E as in (SL) 3 , but with  C trained to generate subtask sequences by maximizing (cid:80)  ,g log  C (  | g ) and  E trained to maximize (cid:80) a , o ,  ,g log  E ( a | o ,  , g ) using only D ann . Because  E maps from complete task sequences to complete low-level action sequences, training of this model involves no explicit alignment or segmentation steps. no-pretrain , no-latent : Ablations of the full (SL) 3 model in which  C and  E are, respectively, randomly initialized or updated only on L ann (  ,   ) during the parameter update phase. We additionally contextualize our approach by comparing it to several state-of-the-art models for the instruction following task in ALFRED: S+ (Shridhar et al., 2020), MOCA (Singh et al., 2020), Modular (Corona et al., 2021), HiTUT (Zhang and Chai, 2021), ABP (Kim et al., 2021), ET (Pashe-vich et al., 2021), EmBERT (Suglia et al., 2021), and FILM (Min et al., 2021). Like seq2seq , these 1718 are neural sequence-to-sequence models trained to map instructions to actions; they incorporate several standard modeling improvements from the instruction following literature, including progress monitoring (Ma et al., 2019) and pretrained object recognizers (Singh et al., 2020). Many of these models are trained with stronger supervision than (SL) 3 , including instructions and alignments during training, and ground truth instructions during evaluation ; see Table 3 for details. Evaluation Following Shridhar et al. (2020), Table 1(a) computes the online , subtask-level accuracy of each policy, and Table 1(b) computes the end-to-end success rate of each policy. See the ALFRED paper for details of these evaluations. For data-efficiency experiments involving a large number of policy variants (Table 2, Fig. 4), we instead use an offline evaluation in which we measure the fraction of subtasks in which a policy's predicted actions (ignoring object selection masks) exactly match the ground truth action sequence. Table 1 compares (SL) 3 with flat and hierarchical imitation learning baselines. The table includes two versions of the model: a 100% model trained with full instruction supervision ( |D| = 0 , |D ann | = 21000 ) and a 10% model trained with only a small fraction of labeled demonstrations ( |D| = 19000 , |D ann | = 2000 ). seq2seq and seq2seq2seq models are always trained with 100% of natural language annotations. Results are shown in Table 1. We find: (SL) 3 improves on flat policies: In both the 10% and 100% conditions, it improves over the subtask completion rate of the seq2seq (goals-to-actions) model by 25%. When either planneror mapping-based navigation is used in conjunction with (SL) 3 , it achieves end-to-end performance comparable to the HLSM method, which relies on similar supervision. Strikingly, it outperforms several recent methods with access to even more detailed information at training or evaluation time. Language-based policies can be trained with sparse natural language annotations: Performance of (SL) 3 trained with 10% and 100% natural language annotations is similar (and in both cases superior to seq2seq and seq2seq2seq trained on 100% of data). Appendix Fig. 4 shows more detailed supervision curves. Ablation experiments in Table 2 show that inference of latent training plans is important for this result: with no inference of (a) Online subtask success rate for (SL) 3 and baselines Model A vg C l e an C oo l H e a t P i ck P u t S li ce T ogg l e G o T o (SL) 3 (10%) 50 56 75 74 50 48 54 32 13 (SL) 3 (100%) 53 68 82 75 50 45 55 32 15 seq2seq 25 16 33 64 20 15 25 13 14 seq2seq2seq 39 15 69 58 29 42 50 32 15 (b) End-to-end task success rates for (SL) 3 and other models. latent instructions (i.e. training only on annotated demonstrations), performance drops from 56% to 52%. Fig. 3 shows an example of the structure inferred for an unannotated trajectory: the model inserts reasonable segment boundaries and accurately labels each step. Language model pretraining improves automated decision-making. Ablation experiments in Table 2 provide details. Language model pretraining of  C and  E (on ungrounded text) is crucial for good performance in the low-data regime: with 10% of annotations, models trained from scratch complete 49% of tasks (vs 56% for pretrained mod-els). We attribute this result in part to the fact that pretrained language models encode information about the common-sense structure of plans, e.g. the fact that slicing a tomato first requires finding a knife . Such models are well-positioned to adapt to planning problems that require modeling relations between natural language strings. These 1719 Figure 3: Example of an inferred segmentation and labeling for an unannotated trajectory. experiments point to a potentially broad role for pretrained language models in tasks that do not involve language as an input or an output. One especially interesting consequence of the use of language-based skills is our model's ability to produce high-level plans for out-of-distribution goals , featuring objects or actions that are not part of the ALFRED dataset at all. Examples are provided in Fig. 5 and discussed in Appendix A. While additional modeling work is needed to generate low-level actions for these high-level plans, they point to generalization as a key differentiator between latent language policies and ordinary hierarchical ones. Our approach draws on a large body of research at the intersection of natural language processing, representation learning, and autonomous control. representation The use of natural language annotations to scaffold learning, especially in computer vision and program synthesis applications, has been the subject of a number of previous studies (Brana-van et al., 2009; Frome et al., 2013; Andreas et al., 2018; Wong et al., 2021). Here, we use language to support policy learning, specifically by using natural language instructions to discover compositional subtask abstractions that can support autonomous control. Our approach is closely related to previous work on learning skill libraries from policy sketches (Andreas et al., 2017; Shiarlis et al., 2018); instead of the fixed skill inventory used by policy sketches, (SL) 3 learns an open-ended, compositional library of behaviors indexed by natural language strings. Hierarchical policies Hierarchical policy learning and temporal abstraction have been major areas of focus since the earliest research on reinforcement learning and imitation learning (McGovern and Barto, 2001; Konidaris et al., 2012; Daniel et al., 2012). Past work typically relies on direct supervision or manual specification of the space of high-level skills (Sutton et al., 1999; Kulkarni et al., 2016) or fully unsupervised skill discovery (Dietterich, 1999; Bacon et al., 2017). Our approach uses policy architectures from this literature, but aims to provide a mechanism for supervision that allows fine-grained control over the space of learned skills (as in fully supervised approaches) while requiring only small amounts of easy-to-gather human supervision. intersection of language and control include instruction following (Chen and Mooney, 2011; Branavan et al., 2009; Tellex et al., 2011; Anderson et al., 2018; Misra et al., 2017), embodied question answering (Das et al., 2018; Gordon et al., 2018) and dialog tasks (Tellex et al., 2020). As in our work, representations of language learned from large text corpora facilitate grounded language learning (Shridhar et al., 2021), and interaction with the environment can in turn improve the accuracy of language generation (Zellers et al., 2021); future work might extend our framework for semi-supervised inference of plan descriptions to these settings as well. We have presented (SL) 3 , a framework for learning hierarchical policies from demonstrations sparsely annotated with natural language descriptions. Using these annotations, (SL) 3 infers the latent structure of unannotated demonstrations, automatically segmenting them into subtasks and labeling each subtask with a compositional description. Learning yields a hierarchical policy in which natural language serves as an abstract representation of subgoals and plans: a controller sub-policy maps from goals to natural language plan specifications, and a modular executor that maps each component of the plan to a sequence of low-level actions. In simulated household environments, this model can complete abstract goals (like slice a tomato ) with accuracy comparable to state-of-the-art models trained and evaluated with fine-grained plans ( find a knife , carry the knife to the tomato , . . . ). While our evaluation has focused on household robotics tasks, the hierarchical structure inferred by (SL) 3 is present in a variety of learning problems, including image understanding, program synthesis, and language generation. In all those domains, generalized versions of (SL) 3 might offer a framework for building high-quality models using only a small amount of rich natural language supervision. We would like to thank Valts Blukis and Shikhar Murty for helpful discussions. Also thanks to Joe O' Connor, Gabe Grand and the anonymous reviewers for their feedback on an early draft of the paper.
method objective method abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain result method objective result other objective abstain result method other other abstain method other abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method method abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method method abstain other abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain method abstain method abstain abstain method method abstain method method abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain other abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain other other