Vision and language navigation (VLN) is a challenging visually-grounded language understanding task. Given a natural language navigation instruction, a visual agent interacts with a graph-based environment equipped with panorama images and tries to follow the described route. Most prior work has been conducted in indoor scenarios where best results were obtained for navigation on routes that are similar to the training routes, with sharp drops in performance when testing on unseen environments. We focus on VLN in outdoor scenarios and find that in contrast to indoor VLN, most of the gain in outdoor VLN on unseen data is due to features like junction type embedding or heading delta that are specific to the respective environment graph, while image information plays a very minor role in generalizing VLN to unseen outdoor areas. These findings show a bias to specifics of graph representations of urban environments, demanding that VLN tasks grow in scale and diversity of geographical environments. 1 1 Introduction Vision and language navigation (VLN) is a challenging task that requires the agent to process natural language instructions and ground them in a visual environment. The agent is embodied in the environment and receives navigation instructions. Based on the instructions, the observed surroundings, and the current trajectory the agent decides its next action. Executing this action changes the position and/or heading of the agent within the environment, and eventually the agent follows the described route and stops at the desired goal location. The most common evaluation metric in VLN is the proportion of successful agent navigations, called task completion (TC). While early work on grounded navigation was confined to grid-world scenarios (MacMahon et al., 2006; Chen and Mooney, 2011), recent work has studied VLN in outdoor environment consisting of real-world urban street layouts and corresponding panorama pictures (Chen et al., 2019). Recent agent models for outdoor VLN treat the task as a sequence-to-sequence problem where the instructions text is the input and the output is a sequence of actions (Chen et al., 2019; Xiang et al., 2020; Zhu et al., 2021b). In contrast to indoor VLN (An-derson et al., 2018; Ku et al., 2020), these works only consider a seen scenario , i.e., the agent is tested on routes that are located in the same area as the training routes. However, studies of indoor VLN (Zhang et al., 2020) show a significant performance drop when testing in previously unseen areas. The main goal of our work is to study outdoor VLN in unseen areas , pursuing the research question of which representations of an environment and of instructions an agent needs to succeed at this task. We compare existing approaches to a new approach that utilizes features based on the observed environment graph to improve generalization to unseen areas. The first feature, called junction type embedding, encodes the number of outgoing edges at the current agent position; the second feature, called heading delta, encodes the agent's heading change relative to the previous timestep. As our experimental studies show, representations of full images do not contribute very much to successful VLN in outdoor scenarios beyond these two features. One reason why restricted features encoding junction type and heading delta are successful in this task is that they seem to be sufficient to encode peculiarities of the graph representation of the environments. Another reason is the current restriction of outdoor environments to small urban areas. In our case, one dataset is the widely used Touchdown dataset introduced by Chen et al. (2019), the 7519 other dataset is called map2seq and has recently been introduced by Schumann and Riezler (2021). The map2seq dataset was created for the task of navigation instructions generation but can directly be adopted to VLN. We conduct a detailed analysis of the influence of general neural architectures, specific features such as junction type or heading delta, the role of image information and instruction token types, to outdoor VLN in seen and unseen environments on these two datasets. Our specific findings unravel the contributions of these features on several VLN subtasks such as orientation, directions, stopping. Our general finding is that current outdoor VLN suffers a bias towards urban environments and to artifacts of their graph representation, showing the necessity of more diverse datasets and tasks for outdoor VLN. Our main contributions are the following:  We describe a straightforward agent model that achieves state-of-the-art task completion and is used as a basis for our experiments. We introduce the unseen scenario for outdoor VLN and propose two environment-dependent features to improve generalization in that setting. We compare different visual representations and conduct language masking experiments to study the effect in the unseen scenario. We adopt the map2seq dataset to VLN and show that merging it with Touchdown improves performance on the respective test sets. The goal of the agent is to follow a route and stop at the desired target location based on natural language navigation instructions. The environment is a directed graph with nodes v  V and labeled edges ( u, v )  E . Each node is associated with a 360  panorama image p and each edge is labeled with an angle  ( u,v ) . The agent state s  S consists of a node and the angle at which the agent is heading: ( v,  ( v,u ) | u  N outv ) , where N outv are all outgoing neighbors of node v . The agent can navigate the environment by performing an action a  { FORWARD , LEFT , RIGHT , STOP } at each timestep t . The FORWARD action moves the agent from state ( v,  ( v,u ) ) to ( u,  ( u,u  ) ) , where ( u, u  ) is the edge with an angle closest to  ( v,u ) . The RIGHT and LEFT action rotates the agent towards ( Instructions Encoder Attention Visual Encoder Attention 1stRNN O bservation 2ndRNN FFNNR ecurrent R ecurrent A ttention I n i t i a li z e RNN c e ll Encoder Decoder Junction Type Step Count Prev. the closest edge angle in clockwise or counterclockwise direction, respectively: ( v,  ( v,u  ) ) . Given a starting state s 1 and instructions text x , the agent performs a series of actions a 1 , ..., a T until the STOP action is predicted. If the agent stops within one neighboring node of the desired target node (goal location), the navigation was successful. The described environment and location finding task was first introduced by (Chen et al., 2019) and we will also refer to it as "outdoor VLN task" throughout this paper. In this section we introduce the model that we use to analyze navigation performance in the unseen and seen scenario for outdoor VLN. The architecture is inspired by the cross-modal attention model for indoor VLN (Krantz et al., 2020). First we give a high level overview of the model architecture and rough intuition. Afterwards we provide a more 7520 formal description. As depicted in Figure 1, the model follows a sequence-to-sequence architecture where the input sequence is the navigation instructions text and the output is a sequence of agent actions. At each decoding timestep, a new visual representation of the current agent state within the environment is computed, where the agent state is dependent on the previously predicted actions. The decoder RNN has two layers where the first encodes metadata and a visual representation. The second RNN layer encodes a contextualized text and visual representation and eventually predicts the next action. The intuition behind the model architecture is to firstly accumulate plain observations available at the current timestep and entangle them with previous observations in the first recurrent layer. Based on these observations, the model focuses attention to certain parts of the instructions text and visual features which are again entangled in the second recurrent layer. Thus, we use the acronym ORAR (observation-recurrence attention-recurrence) for the model. In detail, the instructions encoder embeds and encodes the tokens in the navigation instructions sequence x = x 1 , ..., x L using a bidirectional LSTM (Graves et al., 2005):  x i = embedding ( x i ) (( w 1 , ..., w L ) , z wL ) = Bi-LSTM ( x 1 , ...,  x L ) , where w 1 , ..., w L are the hidden representations for each token and z wL is the last LSTM cell state. The visual encoder, described in detail below, emits a fixed size representation  p t of the current panorama view and a sequence of sliced view representations  p 1 t , ...,  p St . The state z first 0 of the cell in the first decoder LSTM layer is initialized using z wL . The input to the first decoder layer is the concatenation (  ) of visual representation  p t , previous action embedding  a t  1 , junction type embedding  n t , and heading delta d t . The output of the first decoder layer, h firstt = LSTM first ([ a t  1   n t  d t   p t ]) , is then used as the query of multi-head attention (Vaswani et al., 2017) over the text encoder. The resulting contextualized text representation c wt is then used to attend over the sliced visual representations: c wt = MultiHeadAttention ( h firstt , ( w 1 , ..., w L )) c pt = MultiHeadAttention ( c wt , ( p 1 t , ...,  p St )) . h secondt = LSTM second ([ t  h firstt  c wt  c pt ]) , where  t is the embedded timestep t . The hidden representation h secondt of the second decoder LSTM layer is then passed through a feed forward network to predict the next agent action a t . At each timestep t the panorama at the current agent position is represented by extracted visual features. We slice the panorama into eight projected rectangles with 60  field of view, such that one of the slices aligns with the agent's heading. This centering slice and the two left and right of it are fed into a ResNet pretrained 2 on ImageNet (Russakovsky et al., 2015). We consider two variants of ResNet derived panorama features. One variant extracts low level features from the fourth to last layer ( 4th-to-last ) of a pretrained ResNet-18 and concatenates each slice's feature map along the width dimension, averages the 128 CNN filters and cuts out 100 dimensions around the agents heading. This results in a feature matrix of 100  100 (  p 1 t , ...,  p 100 t ). The full procedure is described in detail in Chen et al. (2019) and Zhu et al. (2021b). The other variant extracts high level features from a pretrained ResNet-50's pre-final layer for each of the 5 slices:  p 1 t , ...,  p 5 t . Each slice vector  p st is of size 2 , 048 resulting in roughly the same number of extracted ResNet features for both variants, making a fair comparison. Further, we use the semantic segmentation representation of the panorama images. We employ omnidirectional semantic segmentation (Yang et al., 2020) to classify each pixel by one of the 25 classes of the Mapillary Vistas dataset (Neuhold et al., 2017). The classes include e.g. car, truck, traffic light, vegetation, road, sidewalk. See Figure 1 bottom right for a visualization. Each panorama slice (  p 1 t , ...,  p 5 t ) is then represented by a 25 dimensional vector where each value is the normalized area covered by the corresponding class (Zhang et al., 2020). For either feature extraction method, the fixed sized panorama representation  p t is computed by concatenating the slice features  p 1 t , ...,  p St and passing them to a feed forward network. 2 https://pytorch.org/vision/0.8/models. The junction type embedding is a feature that we introduce to better analyze generalization to unseen areas. It embeds the number of outgoing edges of the current environment node and is categorized into {2, 3, 4, >4}. It provides the agent information about the type of junction it is positioned on: a regular street segment, a three-way intersection, a four way intersection or an intersection with more than four outgoing streets. We want to point out that the number of outgoing edges isn't oracle information in the environment described in Section 2. The agent can rotate left until the same panorama view is observed and thus counting the number of outgoing edges by purely interacting with the environment. But it is clear that the feature leverages the fact that the environment is based on a graph and it would not be available in a continuous setting (Krantz et al., 2020). As described in Section 2, the environment defined and implemented by Chen et al. (2019) only allows states where the agent is heading towards an outgoing edge. As a consequence the environment automatically rotates the agent towards the closest outgoing edge after transitioning to a new node. The environment behavior is depicted in Figure 2a) for a transition between two regular street segments. However, as depicted in Figure 2b), a problem arises when the agent is walking towards a three-way intersection. The automatic rotation introduces unpredictable behavior for the agent and we hypothesis that it hinders generalization to unseen areas. To correct for this environment artifact, we introduce the heading delta feature d t which encodes the change in heading direction relative to the previous timestep. The feature is normalized to (  1 , 1] where a negative value indicates a left rotation and a positive value indicates a right rotation. The magnitude signals the degree of the rotation up to 180  . We use the Touchdown (Chen et al., 2019) and the map2seq (Schumann and Riezler, 2021) datasets in our experiments. Both datasets contain human written navigation instructions for routes located in the same environment. The environment consists of 29,641 panorama images from Manhattan and the corresponding connectivity graph. The Touchdown dataset (Chen et al., 2019) for vision and language navigation consists of 9,326 routes paired with human written navigation instructions. The annotators navigated the panorama environment based on a predefined route and wrote down navigation instructions along the way. The map2seq (Schumann and Riezler, 2021) dataset was created for the task of navigation instructions generation. The 7,672 navigation instructions were written by human annotators who saw a route on a rendered map, without the corresponding panorama images. The annotators were told to include visual landmarks like stores, parks, churches, and other amenities into their instructions. A different annotator later validated the written navigation instructions by using them to follow the described route in the panorama environment (without the map). This annotation procedure allows us to use the navigation instructions in the map2seq dataset for the vision and language navigation task. We are the first to report VLN results on this dataset. Despite being located in the same environment, the routes and instructions from each dataset differ in multiple aspects. The map2seq instructions typically include named entities like store names, while Touchdown instructions focus more on visual features like the color of a store. Both do not include street names or cardinal directions and are written in egocentric perspective. Further, in map2seq the agent starts by facing in the correct direction, while in Touchdown the initial heading is random and the first part of the instruction is about orientating the agent ("Turn around such that the scaffolding is on your right"). A route in map2seq includes a minimum of three intersections and is the shortest path from the start to the end location. 3 In Touchdown there are no such constraints and a route can almost be circular. The routes in both datasets are around 35-45 nodes long with some shorter outliers in Touchdown. On average instructions are around 55 tokens long in map2seq and around 89 tokens long in Touchdown. We are interested in the generalization ability to unseen areas and how it is influenced by the two proposed features, types of visual representation, navigation instructions and training set size. Alongside of the results in the unseen scenario, we report results in the seen scenario to interpret performance improvements in relation to each other. All experiments 4 are repeated ten times with different random seeds. The reported numbers are the average over the ten repetitions. Results printed in bold are significantly better than non-bold results in the same column. Significance was established by a paired t-test 5 on the ten repetition results and a p-value  0 . 05 without multiple hypothesis corrections factor. Individual results can be found in the Appendix. To be able to compare our model with previous work, we use the original training, development and test split (Chen et al., 2019) for the seen scenario on Touchdown. Because we are the first to use the map2seq data for VLN we create a new split for it. The resulting number of instances can be 3 The shortest path bias reduces the number of reasonable directions at each intersection and thus makes the task easier. 4 Except comparison models on the Touchdown seen test set for which we copy the results from the respective work. seen in the left column of Table 1. For the unseen scenario, we create new splits for both datasets. We separate the unseen area geographically by drawing a boundary across lower Manhattan (see Figure 3). Development and test instances are randomly cho-sen from within the unseen area. Routes that are crossing the boundary are discarded. The right column of Table 1 shows the number of instances for both splits. Additionally, we merge the two datasets for both scenarios. This is possible because both datasets are located in the same environment and the unseen boundary is equivalent. We train the models with Adam (Kingma and Ba, 2015) by minimizing cross entropy loss in the teacher forcing paradigm. We set the learning rate to 5e-4, weight decay to 1e-3 and batch size to 64. After 150 epochs we select the model with the best shortest path distance (SPD) performance on the development set. We apply dropout of 0.3 after each dense layer and recurrent connection. The multi-head attention mechanism is regularized 7523 Seen Unseen Touchdown map2seq Touchdown map2seq dev test dev test dev test dev test Model nDTW TC nDTW TC nDTW TC nDTW TC nDTW TC nDTW TC nDTW TC nDTW TC RConcat 22.5 10.6 22.9 11.8 30.7 17.1 27.7 14.7 3.9 2.3 3.5 1.9 3.7 2.0 3.8 2.1 GA 25.2 12.0 24.9 11.9 33.0 18.2 30.1 17.0 3.6 1.8 4.0 2.2 3.9 1.8 4.1 1.7 ARC 15.3 14.1 ------ARC+l2s 19.5 16.7 ------VLN Transformer 23.0 14.0 25.3 14.9 31.1 18.6 29.5 17.0 4.7 2.3 5.2 3.1 6.2 3.6 6.1 3.5 ORAR full model  ResNet pre-final 38.9 26.0 38.4 25.3 65.0 49.1 62.3 46.7 13.0 9.6 12.1 8.8 34.6 24.2 34.5 24.6  ResNet 4th-to-last 45.1 29.9 44.9 29.1 60.0 43.4 57.8 41.7 22.2 15.4 21.6 14.9 41.0 27.6 42.2 30.3 ORAR full model  ResNet 4th-to-last  ResNet pre-final  ResNet 4th-to-last  ResNet 4th-to-last no heading delta 45.5 30.0 45.3 29.3 63.2 47.7 60.3 44.9 21.6 15.2 21.2 14.8 33.0 22.0 33.6 23.6 no junction type 40.6 25.9 40.9 25.5 65.9 52.9 62.1 47.5 7.9 4.8 7.1 4.3 13.1 7.4 11.8 7.1 no head. by attention dropout of 0.3 and layer normalization. The navigation instructions are lower-cased and split into byte pair encodings (Sennrich et al., 2016) with a vocabulary of 2,000 tokens and we use BPE dropout (Provilkov et al., 2020) during training. The BPE embeddings are of size 32 and the bidirectional encoder LSTM has two layers of size 256. The feed forward network in the visual encoder consists of two dense layers with 512 and 256 neurons, respectively, and 64 neurons in case of using semantic segmentation features. The embeddings that encode previous action, junction type, and step count are of size 16. The two decoder LSTM layers are of size 256 and we use two attention heads. Training the full model takes around 3 hours on a GTX 1080 Ti. We compare the ORAR model to previous works. Because these works only report results for the seen scenario on Touchdown, we evaluate those for which we could acquire the code, on the map2seq dataset and the unseen scenario. The models RConcat (Mirowski et al., 2018; Chen et al., 2019), GA (Chaplot et al., 2018; Chen et al., 2019) and ARC (Xiang et al., 2020) use an LSTM to encode the instructions text and a single layer decoder LSTM to predict the next action. They differ in how the text and image representations are incorporated during each timestep in the decoder. As the name suggests, in RConcat the two representations are concatenated. GA uses gated attention to compute a fused representation of text and image. ARC uses the hidden representation of the previous timestep to attend over the instructions text. This contextualized text representation is then concatenated to the image representation. They further introduce ARC+l2s which cascades the action prediction into a binary stopping decision and a subsequent direction classification. The VLN-Transformer (Zhu et al., 2021b) uses pretrained BERT (Devlin et al., 2019) to encode the instructions and VLN-BERT (Majumdar et al., 2020) to fuse the modalities. We use task completion ( TC ) as the main performance metric. It represents the percentage of successful agent navigations (Chen et al., 2019). We further report normalized Dynamic Time Warping ( nDTW ) which quantifies agent and gold trajectory overlap for all routes (Ilharco et al., 2019). The shortest path distance ( SPD ) is measured within the environment graph from the node the agent stopped to the goal node (Chen et al., 2019). The two upper sections of Table 2 show the results of the ORAR model introduced in Section 3 in comparison to other work. While the model sig-7524 Unseen Touchdown map2seq Visual Features dev test dev test ResNet pre-final 9.6 8.8 24.2 24.6 no junction type 4.4 4.0 10.7 11.0 ResNet 4th-to-last 15.4 14.9 27.6 30.3 no junction type 4.8 4.3 7.4 7.1 semantic segmentation 11.5 11.0 29.0 31.1 no junction type 5.5 5.5 11.6 12.1 no image 11.5 9.5 28.5 30.5 no junction type 3.0 2.8 5.4 5.5 Table 3: Study of visual features for the unseen scenario of Touchdown and map2seq. nificantly outperforms all previous work on both datasets, our main focus is analyzing generalization to the unseen scenario. It is apparent that the type of image features influences agent performance and will be discussed in the next section. The bottom section of Table 2 ablates the proposed heading delta and junction type features for the best models. Removing the heading delta feature has little impact in the seen scenario, but significantly reduces task completion in the unseen scenario of the map2seq dataset. Surprisingly, the feature has no impact in the unseen scenario of Touchdown. We believe this is a consequence of the different data collection processes. Touchdown was specifically collected for VLN and annotators navigated the environment graph, while map2seq annotators wrote instructions only seeing the map. Removing the junction type embedding leads to a collapse of task completion in the unseen scenario on both datasets. This shows that without this explicit feature, the agent lacks the ability to reliably identify intersections in new areas. Table 3 shows results for different types of visual features in the unseen scenario. We compare high level ResNet features (pre-final), low level ResNet features (4th-to-last), semantic segmentation features and using no image features. For the ResNet based features, the low level 4th-to-last features perform better than pre-final on both datasets. On map2seq the no image baseline performs on par with models that have access to visual features. When we remove the junction type embedding, the task completion rate drops significantly, which shows that the agent is not able to reliably locate intersections from any type of visual features. The agent has to predict a sequence of actions in order to successfully reach the goal location. In Touchdown this task can be divided into three subtasks (see Section 4). First the agent needs to orientate itself towards the correct starting heading. Next the agent has to predict the correct directions at the intersections along the path. The third subtask is stopping at the specified location. Providing oracle actions (during testing) for two of the three sub-tasks lets us look at the completion rate of the remaining sub-task. Table 4 shows the completion rates for each of the three sub-tasks when using ResNet pre-final, 4th-to-last and no image features. In the seen scenario we can observe that the pre-final features lead to the best performance for the directions task. The 4th-to-last features on the other hand lead to the best orientation task performance and the stopping task is not influenced by the choice of visual features. In the unseen scenario 4th-to-last features again provide best orientation task performance but no image features lead to the best performance for the directions task. This shows that the ResNet 4th-to-last features are primarily useful for the orientation sub-task and explains the discrepancy of the no image baseline on Touchdown and map2seq identified in the previous subsection. In the Appendix we use this knowledge to train a mixed-model that uses 4th-to-last features 7525 0% 20% 40% 60% 80% 100% 0 5 10 15 20 25 30 T a s k C o m p l e t i o n Touchdown Seen 0% 20% 40% 60% 80% 100% Token Masking Percentage 0 5 10 15 20 25 30 T a s k C o m p l e t i o n Touchdown Unseen Mask Object Tokens: Mask Direction Tokens: 4th-to-last 4th-to-last no image no image Figure 4: Masking experiments on the seen and unseen test set of Touchdown. for the orientation sub-task and pre-final/no image features for directions and stopping. To analyze the importance of direction and object tokens in the navigation instructions, we run masking experiments similar to Zhu et al. (2021a), except that we mask the tokens during training and testing instead of during testing only. Figure 4 shows the resulting task completion rates for an increasing number of masked direction or object tokens. From the widening gap between masking object and direction tokens, we can see that the direction tokens are more important to successfully reach the goal location. Task completion nearly doesn't change when masking object tokens, indicating that they are mostly ignored by the model. While task completion significantly drops when direction tokens are masked, the agent still performs on a high level. This finding is surprising and in dissent with Zhu et al. (2021a) who report that task completion nearly drops to zero when masking direction tokens during testing only. We believe that in our setting (masking during testing and train-ing), the model learns to infer the correct directions from redundancies in the instructions or context around the direction tokens. Besides the general trend of lower performance on the unseen scenario, we can not identify different utilization of object or direction tokens in the seen and unseen scenario. We train the ORAR full model on the merged dataset (see Section 5.1). Model selection is performed on the merged development set but results are also reported for the individual test sets of Touchdown and map2seq. For comparison with models trained on the non-merged datasets, the first row of Table 5 shows the best results of Table 2. Training on the merged dataset significantly improves nDTW and task completion across both datasets and scenarios. This shows that both datasets are compatible and the merged dataset can further be used by the VLN community to evaluate their models on more diverse navigation instructions. Despite being trained on twice as many instances, the no image baseline still performs on par on map2seq unseen. From this we conclude that the current bottleneck for better generalization to unseen areas is the number of panorama images seen during training instead of number of instructions. Natural language instructed navigation of embodied agents has been studied in generated grid environments that allow a structured representation of the observed environment (MacMahon et al., 2006; Chen and Mooney, 2011). Fueled by the advances in image representation learning (He et al., 2016), the environments became more realistic by using real-world panorama images of indoor locations (Anderson et al., 2018; Ku et al., 2020). Complementary outdoor environments contain street level panoramas connected by a real-world street layout (Mirowski et al., 2018; Chen et al., 2019; Mehta et al., 2020). Agents in this outdoor environment are trained to follow human written navigation instructions (Chen et al., 2019; Xiang et al., 2020), instructions generated by Google Maps (Hermann et al., 2020), or a combination of both (Zhu et al., 2021b). Recent work focuses on analyzing the navigation agents by introducing better trajectory overlap metrics (Jain et al., 2019; Ilharco et al., 2019) or diagnosing the performance under certain constraints such as uni-modal inputs (Thomason et al., 2019) and masking direction or object tokens (Zhu et al., 2021a). Other work used a trained VLN agent to evaluate automatically generated navigation instructions (Zhao et al., 2021). An open problem in indoor VLN is 7526 Seen Unseen Merged Touchdown map2seq Merged Touchdown map2seq dev test test test dev test test test Model nDTW TC nDTW TC nDTW TC nDTW TC nDTW TC nDTW TC nDTW TC nDTW TC best non-merged --44.9 29.1 62.3 46.7 --21.6 14.9 42.2 30.3 ORAR full model  no image 37.5 26.6 35.8 24.7 23.0 14.8 58.3 42.1 31.6 22.3 27.0 19.2 16.6 11.7 46.5 33.2  ResNet pre-final 51.3 38.8 49.3 36.8 39.1 27.7 67.3 52.8 28.9 22.0 25.7 20.0 17.4 13.6 41.3 32.1  ResNet 4th-to-last 53.4 37.8 51.8 35.7 46.0 30.1 62.1 45.5 35.7 25.4 33.6 24.2 27.0 19.3 46.1 33.5 Table 5: Results for models trained on the merged dataset. the generalization of navigation performance to previously unseen areas. Proposed solutions include back translation with environment dropout (Tan et al., 2019), multi-modal environment representation (Hu et al., 2019) or semantic segmented images (Zhang et al., 2020). Notably the latter work identifies the same problem in the Touchdown task. We presented an investigation of outdoor vision and language navigation in seen and unseen environments. We introduced the heading delta feature and junction type embedding to correct an artifact of the environment and explicitly model the number of outgoing edges, respectively. Both are helpful to boost and analyze performance in the unseen scenario. We conducted experiments on two datasets and showed that the considered visual features poorly generalize to unseen areas. We conjecture that VLN tasks need to grow in scale and diversity of geographical environments and navigation tasks. The research reported in this paper was supported by a Google Focused Research Award on "Learn-ing to Negotiate Answers in Multi-Pass Semantic Parsing".
abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective abstain objective abstain abstain method abstain result objective result objective objective method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result other other other other other other other other other other method method abstain result abstain other