Vision language navigation is the task that requires an agent to navigate through a 3D environment based on natural language instructions. One key challenge in this task is to ground instructions with the current visual information that the agent perceives. Most of the existing work employs soft attention over individual words to locate the instruction required for the next action. However, different words have different functions in a sentence (e.g., modifiers convey attributes, verbs convey actions). Syntax information like dependencies and phrase structures can aid the agent to locate important parts of the instruction. Hence, in this paper, we propose a navigation agent that utilizes syntax information derived from a dependency tree to enhance alignment between the instruction and the current visual scenes. Empirically, our agent outperforms the baseline model that does not use syntax information on the Room-to-Room dataset, especially in the unseen environment. Besides, our agent achieves the new state-of-the-art on Room-Across-Room dataset, which contains instructions in 3 languages (English, Hindi, and Telugu). We also show that our agent is better at aligning instructions with the current visual information via qualitative visualizations. 1 1 Introduction Vision-Language Navigation defines the task of requiring an agent to navigate through a visual environment based on natural language instructions (Anderson et al., 2018b; Misra et al., 2018; Chen et al., 2019; Jain et al., 2019; Nguyen and Daum III, 2019; Thomason et al., 2020). This task poses several challenges. To complete this task, an embodied agent needs to perceive the surrounding environment, understand the given natural language instructions, and most importantly, ground 1 Code and models: https://github.com/ jialuli-luka/SyntaxVLN Navigation Steps: Parse Tree: Instruction: Walk forward then turn right at the stairs then go down the stairs. (or align) the instruction in the visual scenes. In this paper, we aim to make one step towards grounding natural language instructions with visual environment via syntax-enriched alignment. Recently, several approaches were proposed to solve the Vision-Language Navigation task with better interactions between natural language instructions and visual scenes (Fried et al., 2018; Wang et al., 2019; Landi et al., 2019; Wang et al., 2020a; Huang et al., 2019a; Hu et al., 2019; Majumdar et al., 2020; Ma et al., 2019a; Qi et al., 2020b; Zhu et al., 2020a,c). Some approaches utilize soft attention over individual words for better cross-modal grounding, while others improve co-grounding with better language and vision representation and additional alignment module. Although these models achieve significant improvement in performance, they do not explicitly consider syntactic linguistic information in their alignment and decision-making. We argue that the syntactic information (e.g., phrases, word functions, modifiers) captured by dependency parse trees is crucial for accurate alignment between the instructions and the environment. As shown in Figure 1, for the instruction Walk forward then turn right at the stairs then go down the stairs." , the dependency parse tree effectively aggregates syntactically-close words together for the agent (e.g., groups phrase information Walk forward" at node Walk ), and each phrase here corresponds to one navigation action. Besides, the dependency tree structure also helps identify modifiers like for-ward and right . This syntactic information helps the agent identify important words, locate phrases (e.g., sub-instructions), and learn a better alignment between the instruction and the visual environment. Therefore, in this paper, we propose an encoder module that can incorporate simple but important syntactic information from parse trees for the vision-language-navigation task. Our proposed encoder utilizes the ChildSum Tree-LSTM (Tai et al., 2015) over a dependency tree to achieve a syntax-aware representation of the instruction, enabling the agent to focus on more syntactically important words and align not only words but also phrases with the visual scenes. We conduct experiments on both the Room-to-Room (R2R) dataset (Anderson et al., 2018b) and the Room-across-Room (RxR) dataset (Ku et al., 2020). Empirical results show that our proposed approach significantly improves the performance over the baseline model on success rate and achieves the new state-of-the-art (at the time of submission) on the RxR dataset, which contains instruction in three languages (English, Hindi, and Telugu). Moreover, by using structured information from syntax, we are also able to avoid word-level shallow overfit-ting of the model and hence achieve better generalization in the unseen environment. Our analysis further shows that our syntax-aware agent has better interpretability and learns better cross-modality matching. Visual and Textual Grounding in VLN. In vision-language navigation tasks, visual and textual co-grounding aims to learn the relationship between natural language instructions and the visual environments. A main line of research in VLN utilizes soft attention over individual words for cross-modal grounding in both the natural language instruction and the visual scene (Wang et al., 2018, 2019; Tan et al., 2019; Landi et al., 2019; Xia et al., 2020; Wang et al., 2020b,a; Xiang et al., 2020; Zhu et al., 2020b). Other works improve vision and language representations (Hu et al., 2019; Li et al., 2019; Huang et al., 2019b,a; Hao et al., 2020; Majumdar et al., 2020) and propose an additional progress monitor module (Ma et al., 2019b,a; Ke et al., 2019) and object and action aware modules (Qi et al., 2020b) that aid co-grounding. The closest work to ours is from Hong et al. (2020), where they use the dependency tree to generate pre-divided sub-instructions, and then propose a shifting module to select and attend to a single sub-instruction at each time-step, which implicitly captures some syntax information. Compared with them, we directly use the dependency tree to explicitly incorporate syntactic information and get syntax-aware instruction representations, hence achieving substantial improvement in generalizing to the unseen environment. Tree-based Language Representations. Dependency tree provides essential syntactic information for understanding a sentence. Tree-LSTM (Tai et al., 2015) has been widely used to encode parsed tree information and shown improvement over multiple tasks, such as relation extraction (Miwa and Bansal, 2016; Geng et al., 2020), machine translation (Su et al., 2020; Choi et al., 2017; Eriguchi et al., 2016), dialogue (Rao et al., 2019), and language inference (Chen et al., 2017). We are novel in incorporating a dependency tree into the vision-language navigation task via a Tree-LSTM for better phrase-level alignment between the visual environment and language instructions. As illustrated in Figure 2, our base model follows the sequence-to-sequence architecture of previous VLN agents. Our tree-based encoder module is built on top of the strong Environment Drop Agent (Tan et al., 2019). The main difference is that we employ a tree-based language encoder to encode dependency tree information to allow better language grounding. At each time step, we ground all the encoded nodes (i.e., syntax-aware representations) in the dependency tree with the visual information to get the attended textual representation. Generating Dependency Tree Representation with Tree-LSTM. We first generate the dependency parse tree with Stanford CoreNLP (Manning et al., 2014) for English instructions and Stanza Toolkit (Qi et al., 2020a) for Hindi and Telugu instructions. Since the tree-structure is invariant to the children's order (i.e., switching the order of the children of a node doesn't change the tree structure), directly using a Tree-LSTM over an embedding layer may lose important word order information in the instruction. Thus, here we use a bidirectional LSTM with an embedding layer to generate word representations that preserve sequential information of the instruction. Specifically, given an instruction { w i } li =1 , we generate syntax-aware representation {  u dep i } li =1 as:  w i = Embedding( w i ) (1) u 1 , u 2 , ..., u l = Bi LSTM(  w 1 ,  w 2 , ...,  w l ) (2) {  u dep i } li =1 = Tree LSTM( { u i } li =1 ) (3) Visual Encoder and Navigation Decoder. Given panoramic features { f t,p } 36 p =1 and visual representation { g t,k } Kk =1 for K navigable locations at time step t 2 , we picks the next viewpoint from K navigable locations as: p t ( a t = k ) = Softmax k ( g Tt,k WG (cid:101) h t ) (4) where (cid:101) h t is the context aware hidden states, and WG is learned weight parameter. Specifically, we 2 Details for generating these features are in Appendix. compute the (cid:101) h t as:  t,p = softmax p ( f Tt,p WF (cid:101) h t  1 ) (5) (cid:101) f t = (cid:88) p  t,p f t,p (6) h t = LSTM([ (cid:101) f t ; (cid:101) a t  1 ] , (cid:101) h t  1 ) (7)  t,i = softmax i ( u Tdep i WU h t ) (8) (cid:101) u t = (cid:88) i  t,i  u dep i (9) (cid:101) h t = tanh( WM [ (cid:101) u t ; h t ]) (10) where (cid:101) a t  1 is the previous action embedding, (cid:101) f t is the attended panoramic representation, and h t is the decoder hidden state. WF , WU , WM are learned weight parameters. We compute the attended language representation over all dependency node representations which are aware of syntax information. We use a mixture of imitation learning and reinforcement learning to train the agent. Details can be found in Appendix. We evaluate our agent on Room-to-Room (R2R) dataset (Anderson et al., 2018b) and Room-Across-Room (RxR) dataset (Ku et al., 2020). Both datasets are built on Matterport3D simulator (An-derson et al., 2018b). The R2R dataset contains 21567 human-annotated instructions with an average instruction length of 29. The dataset is divided into training set, seen validation set, unseen validation set, and test set. The RxR dataset is an extension of the R2R dataset. The instructions are longer (with an average instruction length of 78), and the instructions are in three languages (i.e., English, Hindi, and Telugu). The RxR dataset follows the same division as the R2R dataset. Details can be found in the Appendix. To evaluate the performance of our model, we use the following evaluation metrics: Success Rate (SR), Success Rate Weighted by Path Length (SPL) (Anderson et al., 2018a), normalized Dynamic Time Warping (nDTW) (Magalhaes et al., 2019), success rate weighted by Dynamic Time Warping (Magalhaes et al., 2019) and Coverage weighted by Length Score (CLS). Detailed description for each metric can be found in Appendix. We compare our agent with the baseline agent (Tan et al., 2019) 3 on the R2R test leaderboard. Our syntax-aware agent achieves 47.8% in success rate and 0.45 in SPL, improving the baseline model by 2.1% in success rate and 2% in SPL. Besides, as shown in Table 1, our syntax-aware agent achieves 3.7% improvement in success rate over the baseline model in validation unseen environment, indicating that explicitly incorporating syntax information can better guide the agent to ground to visual scenes. We further experiment on R2R dataset to see whether the increase in performance comes from more model parameters. Compared with the model that uses a 2-layer LSTM (+LSTM), we can see that our syntax-aware model still achieves 2.6% increase in success rate in validation unseen environment. This result validates the effectiveness of incorporating syntax information. We first compare our agent with the baseline agent in RxR paper (Ku et al., 2020) on the RxR test leaderboard. Our syntax-aware agent (SAA (mono) on the leaderboard) outperforms the baseline in all metrics, improving the nDTW score by 5.73% and success rate by 9.98%. Moreover, We compare our agent with the baseline on RxR validation set. As shown in Table 2, in all three languages, 3 The baseline is the re-implementation of their model without back translation based on code: https://github.com/airsplay/R2R-EnvDrop our syntax-aware agent achieves higher success rate than the baseline agent in all metrics on validation unseen set. Specifically, our model gets 2.6% improvement in Hindi instructions in terms of success rate. For English and Telugu instructions, our model gets smaller improvement  0.7% and 0.1% respectively. One reason for these results could be the correlation with the quality of the dependency parser in that language. Besides, compared with the baseline in Ku et al. (2020), our agent achieves the new state-of-the-art on RxR dataset (at the time of submission). As shown in Fig 3 and Fig 4, we illustrate a qualitative example to show that our agent learns better alignment between instruction and visual scenes. As shown in Fig 4, given instruction Walk past the shelves and out of the garage. Stop in front of the opening to the wine cellar." , the baseline agent tends to focus on the same phrases (e.g., past the shelves, wine cellar) during navigation. This suggests that the baseline agent is not able to learn a good alignment between the instruction and the current visual scene. However, as shown in Figure 3, our dependency-based agent can successfully identify the correct parts of the instruction that are correlated with the current visual scenes, and picks the next action with fidelity. At the beginning of navigation, our agent focuses on the first sub-instruction walk past the shelves. Then out of the garage" gradually becomes the most important phrase, indicating that the agent should go out of the garage after passing the shelves. When walk past the shelves and out of the garage stop in front of the opening to the wine cellar . the agent sees the wine sculpture, it infers that the opening is for wine cellar and stops in front of the opening near the wine sculpture. Since the goal of our paper is to explore the role of syntax information in vision-language navigation, we try several implementation variants to include syntax information. First, we try to use mean-pooling instead of Tree-LSTM to encode the dependency tree structure. This implementation variant decreases the performance by around 3% in terms of success rate on validation unseen environments. Besides, we explore whether syntax information from a constituency tree can also help with the instruction following and navigation. Similar to how we incorporate dependency tree information, we use a Tree-LSTM to encode the constituency tree information. However, the performance decreases around 2% in terms of success rate in validation unseen environments, indicating that syntax information extracted from dependency tree is more beneficial for the vision-and-language navigation task. In this paper, we presented a tree-based encoder module that incorporates phrase-level information from parse trees. We demonstrated that syntax information can help the agent learn a better alignment between instruction and visual scenes, and generalize better to unseen environments. Our experiments on Room-to-Room dataset and Room-Across-Room dataset both suggest that incorporating syntax information encoded by our tree-based encoder module can significantly improve the performance over baseline VLN models. Vision-Language Navigation is the task that requires an agent to navigate through a 3D environment based on given natural language instructions. An agent that can interact with the environment based on instructions can be used in many real-world applications, for example, a home service robot can bring things to the owner based on instruction, making people's life easier. However, when deployed in the real world, even if the agent can navigate successfully based on the instruction, it might still need further human assistance to keep working successfully (e.g., a home-cleaning robot might be stuck in the corner of the room and cannot get out by itself). The performance on the validation set with the unseen environment is much lower than the seen environment. Change in environment will significantly influence the performance of the agent. When the agent is deployed in an unseen environment, it will have a higher probability of failure, wasting energy and time. A further pre-exploration of the environment will be needed for better performance of the agent when deployed to real-world applications. Moreover, our agent relies on the quality of the dependency parser to some extent. Though we achieve improvement in all three languages when using the dependency tree information, the agent in Hindi and English benefit most from the syntax information because of the best available parser for these languages. We thank the reviewers for their helpful discussions. This work was supported by ARO-YIP Award W911NF-18-1-0336, DARPA MCS Grant N66001-19-2-4031, a Google Focused Research Award, and a Bloomberg Data Science Ph.D. Fellowship. The views, opinions, and/or findings contained in this article are those of the authors and not of the funding agency.
abstain abstain abstain abstain abstain objective result objective result abstain abstain other abstain objective other abstain abstain abstain abstain abstain abstain objective abstain method objective result result other other other other other abstain other other other objective method method method method abstain method abstain method method method method other abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain result result result result abstain abstain result abstain other abstain abstain abstain abstain result other abstain abstain abstain abstain abstain abstain abstain abstain method objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other