Over the last few years, there has been growing interest in learning models for physically grounded language understanding tasks, such as the popular blocks world domain. These works typically view this problem as a singlestep process, in which a human operator gives an instruction and an automated agent is evaluated on its ability to execute it. In this paper we take the first step towards increasing the bandwidth of this interaction, and suggest a protocol for including advice, high-level observations about the task, which can help constrain the agents prediction. We evaluate our approach on the blocks world task, and show that even simple advice can help lead to sig-nificant performance improvements. To help reduce the effort involved in supplying the advice, we also explore model self-generated advice which can still improve results. The problem of constructing an artificial agent capable of understanding and executing human instructions is one of the oldest long-standing AI challenges (Winograd, 1972). This problem has numerous applications in various domains (plan-ning, navigation and assembly) and can help accommodate seamless interaction with personal assistants in many environments. Due to its central role in AI and wide applicability, this problem has seen a surge of interest recently (MacMahon et al., 2006; Branavan et al., 2009; Chen and Mooney, 2011; Tellex et al., 2011; Matuszek et al., 2012; Kim and Mooney, 2013; Misra et al., 2017). Recent works (Bisk et al., 2016; Tan and Bansal, 2018) focus on exploring deep learning methods for grounding spatial language. In this popular setup, human communication with robots is viewed as a single-step process, in which a natural language (NL) instruction is provided, and an outcome is observed. Our goal in this paper is to explore different approaches for relaxing the single step assumption, and present initial results which we hope would motivate future work in this direction. Similar to interactive dialog systems (Allen et al., 1995; Ryb-ski et al., 2007; Wen et al.), we view this problem as an interactive process, in which the human operator can observe the agents' response to their instruction and adjust it by providing advice, a form of online feedback. Specifically, the advice consists of a short sentence, simplifying the user's intent. We utilize two types of advice, one restricting the agent's search space to a general region ( restrictive advice ), and the other telling the agent the appropriate direction (up, down, left, right) to adjust its current prediction ( corrective advice ). Our focus is on the challenging task of moving blocks on a grid (Winograd, 1972), in which the agent is given only an instruction and the state of the grid, and must predict the coordinates of where a block must be moved. We follow the difficult experimental settings suggested by (Bisk et al., 2016), in which the blocks are unlabeled and can only be referenced by their spatial properties. Fig. 1 describes our settings and uses the advice the target is in the lower left , to restrict the agents search space after observing the incorrect prediction placed the target block in the top half of the board. To accommodate these settings, we take a two step approach. First, we ground the advice text in the simulated blocks-world environment by training a neural network. In the second step, we integrate the trained advice network into the end-to-end neural model proposed by Bisk et al. Our architecture is described in Fig. 2. The experiments we run show that this end-to-end advice model successfully grounds the meaning of our advice. We propose four novel interactive advice-based protocols that can be applied on any robot communication architecture, ordered in terms of decreasing human effort. As expected, as human effort lessens, performance does worsen, but all protocols outperform Bisk et al. (whom our model is identical to besides the inclusion of advice). Most notably, we explore the notion of model self-generated advice, which significantly re-duces/eliminates human effort. In this approach, a model is trained to automatically generate restrictive advice for a given scenario, based on the assumption that it is easier to predict a region containing the target coordinates rather than their exact location. We validate this assumption by developing a neural architecture to predict the restrictive advice and show it can help improve the overall prediction quality, despite having no human assistance. This section describes the architecture we developed for understanding advice, and how to incorporate it into the original Bisk et al. model to make better predictions. We begin by defin-ing the Blocks World task and the types of advice we use. We then introduce a model for grounding the advice and a method for incorporating the pre-trained advice understanding module into the original model. Finally, we discuss an architecture for advice generation , a method for self-predicting the advice to avoid any human intervention. Further details of our models and advice generation process are in the Appendix. Given an input state, consisting of all the block positions on a board, and a NL instruction, the model has to predict the coordinates of the source block to be moved and its target location. We follow the definition by (Bisk et al., 2016) and due to space constraints refer the reader to that paper. The two types of advice we devise in this paper are designed to assist the prediction agent by providing simpler instructions in addition to the original input. The first, restrictive advice, informs the agent about the general region of the source / target coordinates, such as top left . These regions are determined by dividing the board into equally sized sections (two halves, four quadrants). The second type of advice, corrective advice, observes the agents' predictions and determines which direction (up, down, left, right) they must be adjusted to get closer to the target. Both of these are representative of information a human could easily provide to a robot in various ways (speech, using assisted devices, etc.), to help correct its predictions. Specific examples are shown below. We pre-train a neural network model to accurately understand the advice. For both types of advice, a LSTM-RNN (Hochreiter and Schmid-huber, 1997) is used to read the advice sentence s = w 1 , w 2 , ..., w n and output the hidden state representations { h n } . Prior to this, a word embedding layer is used to project the input words into high-dimension vectors { w i } . For restrictive advice, the last state from the LSTM h n is fed along with a random coordinate into a Fully Connected (FC) layer. The network must output a positive prediction if the random coordinate is in the region described by the advice sentence, and negative otherwise. For corrective advice, the last state from the LSTM h n is fed along with a random coordinate into a FC layer, and the network must output a coordinate that follows the advice. For example, if the advice is move the block down , the predicted coordinate must be below the random input coordinate. If the advice is followed, the network receives 0 loss, otherwise a MSE regression loss. the best performing End-to-End RNN architecture proposed in (Bisk et al., 2016) by adding a FC layer to the pre-trained LSTM state h n and summing it with the LSTM hidden state of the original model (as shown in Figure 2b). We load and freeze the best performing parameters from our pre-trained model into the relevant portion of this end-to-end architecture, and train it on the original task of predicting the coordinates of the source / target location, with the addition of advice input. We use a neural network model to self-generate restrictive advice (as shown in Figure 3), passing the instruction into an embedding layer followed by a LSTM, the board state into a FC layer, concatenating these into a FC layer, and finally using a softmax to classify the input example into a region. region. We train this architecture and then run it on the test set, generate the appropriate advice based on the region the data is classified in, and use that as test advice input for the end-to-end architecture from section 2.4. Next, we present our experiments over our four different advice protocols, each with decreasing human effort and overall performance. In each protocol, we provide advice to the end-to-end model from Section 2.4, whether it is given by a human user or model self-generated. Our results, evaluated on each model's mean and median prediction error, are presented in Table 1. We always compare to the baseline Bisk et al. model, which our model is identical to besides the addition of advice (and we always beat), and the state-of-the-art best non-ensemble Tan and Bansal architecture. Note that Tan and Bansal use an advanced neural architecture and a different training procedure (source prediction trained as classification). We hypothesize that using the advice mechanism over this more complex architecture would lead to further improvements, and leave it for future work. The pre-trained advice grounding models from Section 2.3 achieve 99.99% accuracy, and are vital, as shown by the poor performance without them ( M4 vs M5 ). These grounding models allow the end-to-end architecture to generalize to the variability in advice utterances. When training the end-to-end model from Section 2.4, we provide restrictive advice at training time for only half the examples. For every epoch, a different half set of examples (determined randomly) receive advice. This mechanism gives the model a chance to learn to interpret each example with and without advice, so that it can handle the interactivity without overfitting to one setup. This setup also gave the best performance. At test time, the advice is provided only whenever the predictions fall in the wrong general region, just like a human would. As seen in Table 1, this model ( M5 ) significantly outperforms both baselines ( M1 , M3 ). We note that the performance did not improve much when advice was always provided, showing that this model was able to perform well in its absence and does not rely on it (due to our choice not to provide advice all the time in training). In fact a human would only have to provide restrictive advice for 395/720 examples, and the model always follows it. 1 3.2 Corrective Advice We train corrective advice identically to restrictive advice from Section 3.1, except we train in two separate iterations. This is necessary as the model must learn to adjust its predictions based on the advice, which is why it is first trained to make the normal prediction (first iteration), then trained to adjust the prediction (second iteration). In the first iteration, we train identically to (Bisk et al., 2016) with no advice, but in the second iteration corrective advice is generated based on which direction the predictions must be adjusted to be more accurate. This case is simpler than restrictive advice, since the human operator just has to provide the direction to adjust the predictions, rather than the precise region of the coordinates. However, the performance does worsen ( M5 vs M6 ). In Section 2.5, we introduced a model that was able to self-generate restrictive advice by predicting the general region of the block coordinates given the NL instruction and blocks world. Table 2 shows this model's accuracy on that task when the board is split into 4 regions. As this is a hard problem with low accuracy ( A1 ), we instead generate advice for the top 2 most confident predictions (determined by the softmax scores) ( A2 ). We now introduce a new multi-step retry advice protocol. In the first step, the model from Section 2.5 self-generates restrictive advice based on the most confident predicted region, which it uses as input in the end-to-end model. If the user believes the coordinate prediction based on this advice is wrong, it can tell the model to retry, and then the second most likely restrictive advice will be used. Thus, the only human feedback needed now is telling the model to retry, rather than accurate advice as before. The performance of this ( M7 ) still significantly outperforms Bisk et al. and is close to Tan and Bansal on target prediction. We now aim to avoid any human interaction, by letting the model completely self-generate the advice. Accomplishing it would allow us to improve the model's performance without additional human effort. We experimented with two approaches. In the first, we generate advice as described in Section 3.3. However, instead of having the user ask the model to retry, we treat the top 2 confidence regions as a general region, and provide that as advice input as described in Section 3.1. In this case, there is a performance improvement over Bisk et al. with no human effort required ( M8 in Table 1). Our second approach for self-generated advice aims to improve on some of the shortcomings of the first approach. Previously, when generating the advice, we had decided on four coarse-grained regions, and trained a model to classify each input example into one of these regions. In many cases, x Self-Generated Advice (a) The Bisk et al. model would have made a prediction (x') close to the true block (square). However, the advice region (blue) was incorrect (due to the true block being close to the edge of it) and this led to a significantly worse prediction (circle). (b) In the input-specific self-generated advice model, the advice region (blue) is centered at the incorrect coordinate prediction (x'), leading to the true source block being included and a correct prediction (circle). the true coordinate lay close to the boundary of one of these regions, often resulting in the model predicting the wrong region when self-generating the advice. This incorrect prediction would lead to significantly worse performance (when compared to the model without advice) when running the end-to-end model from Section 3.1, as the advice was incorrect (remember that the model always follows our advice, and the true coordinate is not in the advice region due to the mistake). However, if we had instead chosen our regions to be centered at the true coordinate of each input example, it would be less likely that the model would make an incorrect region prediction (since a small error would still lead to the region containing the correct coordinate). Figure 4 provides a visual explanation of this. For this reason, we now introduce input-specific model self-generated advice. In this case, we run the Bisk et al. coordinate prediction model in two iterations. In the first iteration, we use the prediction to generate advice for a region (of the same size as in the case of 4 quadrants) centered at the predicted coordinate (see Figure 4b). 2 In the second iteration, we feed in this generated advice just like Section 3.1. This model ( M9 ) achieves performance slightly worse than retry advice, and significantly better than Bisk et al., all with no human effort. 3 Table 2 shows the accuracy increase 2 We make sure the advice region doesn't exceed the board boundaries. in predicting the advice now ( A3 vs A1 ). It is unsurprising that this approach to self-generating advice performs better, as now the regions are more specific to each coordinate (so there is a higher probability that the true coordinate is actually in the predicted region see Figure 4). We hypothesize that the performance improvements in self-generated advice happen since it is easier to predict the general region used to generate the advice rather than the specific coordinates. Previously, we have also shown the benefit of restrictive advice in improving overall coordinate prediction, so it is unsurprising that a high accuracy of advice generation leads to better overall performance. Due to this, we propose that future robot communication works take advantage of predicting and then using model self-generated advice in their end-to-end training procedure. This paper takes a first step towards a stronger interaction between automated agents and their human operators, for physically grounded language understanding tasks. We focus on the popular blocks task and introduce the notion of advice, Natural Language hints provided by the human operator, correcting the model's predictions. We show that using four versions of this interactive advice driven protocol on an existing robot communication architecture, we can obtain signifi-cant performance improvements. The last method, model self-generated advice, shows the benefit of considering advice even when not designing an interactive protocol. Our future work focuses on further increasing the accuracy of the self-generated advice model, so we can achieve better performance with no human effort. We thank the anonymous reviewers of this paper for all of their vital feedback. This work was partially supported by the Defense Advanced Research Projects Agency (DARPA) under the ASED program. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the DARPA. as there are now significantly more regions. The accuracy of that model is still 99.99%, and the training procedure does not change.
abstain abstain objective result objective abstain abstain abstain abstain abstain objective result abstain method method method result method objective abstain abstain result objective result objective abstain objective abstain method method method abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain method abstain abstain abstain other other other other other