Standard architectures used in instruction following often struggle on novel compositions of subgoals (e.g. navigating to landmarks or picking up objects) observed during training. We propose a modular architecture for following natural language instructions that describe sequences of diverse subgoals. In our approach, subgoal modules each carry out natural language instructions for a specific subgoal type. A sequence of modules to execute is chosen by learning to segment the instructions and predicting a subgoal type for each segment. When compared to standard, non-modular sequence-to-sequence approaches on ALFRED (Shridhar et al., 2020), a challenging instruction following benchmark, we find that modularization improves generalization to novel subgoal compositions, as well as to environments unseen in training. Work on grounded instruction following (MacMa-hon et al., 2006; Vogel and Jurafsky, 2010; Tellex et al., 2011; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013) has recently been driven by sequence-to-sequence models (Mei et al., 2016; Hermann et al., 2017), which allow end-to-end grounding of linguistically-rich instructions into equally-rich visual contexts (Misra et al., 2018; Anderson et al., 2018; Chen et al., 2019). These sequence-to-sequence models are monolithic : they consist of a single network structure which is applied identically to every example in the dataset. Monolithic instruction following models typically perform well when evaluated on test data from the same distribution seen during training. However, they often struggle in compositional generalization : composing atomic parts, such as actions or goals, where the parts are seen in training but their compositions are not (Lake and Baroni, 2018; Ruis et al., 2020; Hill et al., 2020). In this work, we improve compositional generalization in instruction following with modular networks , which have been successful in non-embodied language grounding tasks (Andreas et al., 2016; Hu et al., 2017; Cirik et al., 2018; Yu et al., 2018; Mao et al., 2019; Han et al., 2019) and in following synthetic instructions or symbolic policy descriptions (Andreas et al., 2017; Oh et al., 2017; Das et al., 2018). Modular networks split the decision making process into a set of neural modules. Modules are each specialized for some function, composed into a structure specific to each example, and trained jointly to complete the task. because of their composable structure (Devin et al., 2017; Andreas et al., 2017; Bahdanau et al., 2019; Purushwalkam et al., 2019), and that they can generalize to new environments or domains through module specialization (Hu et al., 2019; Blukis et al., 2020). However, all this work has either focused on grounding tasks without a temporal component or used a network structure which is not predicted from language. We propose a modular architecture for embodied vision-and-language instruction following 1 , and find that this architecture improves generalization on unseen compositions of subgoals (such as navigation, picking up objects, cleaning them, etc.). We define separate sequence-to-sequence modules per type of subgoal. These modules are strung together to execute complex high-level tasks. We train a controller to predict a sequence of subgoal types from language instructions, which determines the order in which to execute the modules. We evaluate models on the ALFRED dataset (Shridhar et al., 2020), an instruction-following benchmark containing a diverse set of household tasks. We focus on compositional generalization: carrying out instructions describing novel high-level tasks, containing novel compositions of actions (see Figure 1 for an example). We find that our modular model improves performance on average across subgoal types when compared to a standard, monolithic sequence-to-sequence architecture. Additionally, we find improved generalization to environments not seen in training. We focus on following instructions in embodied tasks involving navigation and complex object interactions, as shown in Figure 2. In training, each set of full instructions ( e.g. Turn right and cross the room ... Place the vase on the coffee table to the left of the computer.) is paired with a demonstration of image observations and actions. In training, we further assume that the full instruction is segmented into subgoal instructions , and each subgoal instruction is labeled with one of a small number (in our work, 8) of subgoal types , e.g. [Walk to the coffee maker.: GOTO ], [Pick up the dirty mug...: PICKUP ], . . . , and paired with the corresponding segment of the demonstration. During evaluation, the agent is given only full instructions (which are unsegmented and unlabeled), and must predict a sequence of actions to carry out the instructions, conditioning on the image observations it receives. Our modular architecture for compositional instruction following consists of a high-level controller (Figure 2, left), and modules for each subgoal type (Figure 2, right). The high-level controller chooses modules to execute in sequence based on the natural language instructions, and each chosen module executes until it outputs a STOP action. The modules all share the same sequence-to-sequence architecture, which is the same as the monolithic architecture. We initialize each module's parameters with parameters from the monolithic model, and then fine-tune the parameters of each module to specialize for its subgoal. Our instruction-based controller is trained to segment a full instruction into sub-instructions and predict the subgoal type for each sub-instruction. We use a linear chain CRF (Lafferty et al., 2001) that conditions on a bidirectional-LSTM encoding of the full instruction and predicts tags for each word, which determine the segmentation and sequence of subgoal types. This model is based on standard neural segmentation and labelling models (Huang et al., 2015; Lample et al., 2016). We train the controller on the ground-truth instruction segmentations and subgoal sequence labels, and in evaluation use the model to predict segmentations and their associated subgoal sequences (Figure 2, top left). This predicted sequence of subgoals determines the order to execute the modules (Figure 2, right). We use a BIO chunking scheme to jointly segment the instruction and predict a subgoal label for each segment. Formally, for a full instruction of length N , the controller defines a distribution over subgoal tags s 1: N for each word given the instruction x 1: N as p ( s 1: N | x 1: N )  exp N (cid:88) n =1 (cid:0) U s n + B s n  1 ,s n (cid:1) The subgoal tag scores U s n for word n are given by a linear projection of bidirectional LSTM features for the word at position n . The tag transition scores B s n  1 ,s n are learned scalar parameters. In training, we supervise s 1: N using the segmentation of the instruction x 1: N into K subgoal instructions and the subgoal label for each instruction. To predict subgoals for a full instruction in evaluation, we obtain arg max s 1: N p ( s 1: N | x 1: N ) using Viterbi decoding, which provides a segmentation into sub-instructions and a subgoal label for each sub-instruction. Our modularized architecture may be seen in Figure 2, right. The architecture consists of 8 independent modules, one for each of the 8 subgoals in the domain ( e.g. GOTO , PICKUP ). For each module, we use the same architecture as Shridhar et al. (2020)'s monolithic model. This is a sequence-to-sequence model composed of an LSTM decoder taking as input an attended embedding of the natural language instruction, pretrained ResNet-18 (He et al., 2016) features of the image observations, and the previous action's embedding. Hidden states are passed between the modules' LSTM decoders at subgoal transitions (Figure 2, right). At each time step, each module M i computes its hidden state based on the last time step's action a t  1 , the current time step's observed image features o t , an attended language embedding  x it , and the previous hidden state h it  1 : e it = [ a t  1 ; o t ;  x it ] h it = LSTM i ( e it , h it  1 ) Each module's attended language embedding  x it is produced using its own attention mechanism over embeddings X = x 1: N of the language instruction, which are produced by a bidirectional LSTM encoder: z it = ( W ix h it  1 ) (cid:62) X  it = Softmax ( z it )  x it = (  it ) (cid:62) X Finally, the action a t and object interaction mask m t are predicted from h it and e it with a linear layer and a deconvolution network respectively. More details about this architecture can be found in Shridhar et al. (2020). Both the action and mask decoders, well as the language encoder, are shared across modules. 2 Our use of subgoal modules is similar to the hierarchical policy approaches of Andreas et al. (2017), Oh et al. (2017), and Das et al. (2018). However, in those approaches, the input to each module is symbolic ( e.g. FIND [ KITCHEN ]). In contrast, all modules in our work condition directly on natural language. We first pre-train the monolithic model by maximizing the likelihood of the ground-truth trajectories in the training data (Shridhar et al., 2020). We train for up to 20 epochs using the Adam optimizer (Kingma and Ba, 2014) with early stopping on validation data (see Appendix A.1 for hyperparam-eters). We use this monolithic model to initialize the parameters of each of the modules, which have identical architecture to the monolithic model, and 2 The modules' instruction encoder is separate from the controller's encoder (Sec. 2.1), as we found it possible to achieve high performance on the subgoal prediction task using a smaller encoder than the one used by the modules. fine-tune them using the same training and early stopping procedure on the same validation data, 3 allowing the monolithic model's parameters to specialize for each module. Each module predicts only the actions for its segment of each trajectory; however, modules are jointly fine-tuned, passing hidden states (and gradients) from module to module. We evaluate models on out-of-domain generalization in two conditions (see below) using the ALFRED benchmark (Shridhar et al., 2020), comparing our modular approach to their non-modular sequence-to-sequence model. ALFRED is implemented in AI2-THOR 2.0 (Kolve et al., 2017), which contains a set of simulated environments with realistic indoor scene renderings and object interactions. The dataset contains approximately 25K expert instruction-trajectory pairs, comprised of 3 instructions for each of 8K unique trajectories. The instructions include both a high level instruction and a sequence of low level instructions. In our experiments, we do not use the high level instructions, which Shridhar et al. (2020) found to produce comparable results when evaluated on generalization to unseen environments with these architectures. Figure 1 shows two example trajectories and their associated instructions. Trajectories are composed (see Sec. 2) of sequences of eight different types of subgoals: navigation (GOTO ) and a variety of object interactions ( e.g. PICKUP , CLEAN , HEAT ). Each subgoal's subtrajectory is composed of a sequence of low-level discrete actions which specify commands for navigation or object interactions (which are accompanied by image segmentations to choose the object to interact with). The ALFRED dataset was constructed to test generalization to novel instructions and unseen environments. However, all evaluation trajectories in the dataset correspond to sequences of subgoals that are seen during training. For example, some training and evaluation instances might both correspond to the underlying subgoal sequence GOTO , PICKUP , GOTO , PUT , but differ in their low-level actions, their language descriptions, and possibly also the environments they are carried out in. Novel Tasks. We evaluate models' ability to generalize to different high-level tasks (compositions of subgoals) than seen in training. The dataset contains seven different task types, such as Pick & Place , as described in Appendix B.1. We hold out two task types and evaluate models on their ability to generalize to them: Pick Two & Place and Stack & Place . These tasks are chosen because they contain subgoal types that are all individually seen in training, but typically in different sequences. We create generalization splits pick-2-seen and pick-2-unseen by filtering the seen and unseen splits below to contain only Pick Two & Place tasks, and remove all Pick Two & Place tasks from the training data. We create splits stack-seen and stack-unseen for Stack & Place similarly. Novel Instructions and Environments This is the standard condition defined in the original ALFRED dataset. There are two held-out validation sets: seen , which tests generalization to novel instructions and trajectories but through environments seen during training, and unseen , which tests generalization to novel environments: rooms with new layouts, object appearances, and furnishings. We compare our modular architecture with the monolithic baseline, averaging performance over models trained from 3 random seeds. For each generalization condition, we measure success rates over full trajectories as well as over each subgoal type independently. Due to the challenging nature of the domain, subgoal evaluation provides finer-grained comparisons than full trajectories. We use the same evaluation methods and metrics as in Shridhar et al. (2020). Success rates are weighted by path lengths to penalize successful trajectories which are longer than the ground-truth demonstration trajectory. To evaluate full trajectories, we measure path completion: the portion of subgoals completed within the full trajectories. To evaluate the subgoals independently, we advance the model along the expert trajectory up until the point where a given subgoal begins (to maintain a history of actions and observations), then require the model to carry out the subgoal from that point. We also report results from Shridhar et al. (2020) and Singh et al. (2020). We note that the approach of Singh et al. (2020) obtains higher performance on full trajectories than the system of Shridhar et al. (2020) (which we base our approach on) Model C l ea n C oo l G o t o H ea t P i c kup P u t S li ce T ogg l e A vg . primarily by introducing a modular object interaction architecture (shared across all subgoals) and a pre-trained object segmentation model. These techniques could also be incorporated into our approach, which uses modular components for individual subgoal types. Novel Tasks. Table 1 shows for each split the success rates on subgoals appearing in at least 50 validation examples. The modular outperforms the monolithic model on both seen and unseen splits (Tables 1b and 1c). Full trajectory results for novel task generalization are shown in Table 2. In the double generalization condition (unseen environments for the held-out pick-2 and stack tasks) on full trajectories, neither model completes subgoals successfully. Overall, we find that modularity helps across most generalization conditions. Generalization to novel environments. We also compare models on generalization to unseen environments. In the independent subgoal evaluation, the monolithic and modular models perform equally on average in the standard-seen split (Ta-Standard Standard Pick-2 Stack Model seen unseen seen seen S+ 9.4 (5.7) 7.4 (4.7)   MOCA 28.5 (22.3) 13.4 (8.3)   Mono. ble 1a, top). However, in the standard-unseen split (Table 1a, bottom), our modular model outperforms the baseline substantially, with an average success rate of 57% compared to the monolithic model's 46% . (On subgoal types not shown, the modular model still outperforms the monolithic, by margins up to 16%.) In the full trajectory results (Table 2) we see comparable performance between the monolithic and modular models on unseen environments. We introduced a novel modular architecture for grounded instruction following where each module is a sequence-to-sequence model conditioned on natural language instructions. With the ALFRED dataset as a testbed, we showed that our modular model achieves better out-of-domain generalization, generalizing better at the subgoal level to novel task compositions and unseen environments than the monolithic model used in prior work. All of the module types in our model currently use separate parameterizations but identical architectures; future work might leverage the modularity of our approach by using specialized architectures, training procedures, or loss functions for each subgoal type. Furthermore, unsupervised methods for jointly segmenting instructions and trajectories without requiring labeled subgoal labels and alignments would be a valuable addition to our framework. This material is based upon work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE 1752814, a Ford Foundation fellowship to the first author, a Google PhD fellowship to the second author, and by DARPA through the XAI program and the LwLL program.
abstain objective abstain abstain abstain other abstain abstain abstain result abstain abstain abstain method objective method abstain method method objective result result result abstain method abstain method abstain abstain method method method abstain method abstain method other abstain method result abstain abstain method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain result objective objective method abstain other other