Vision-language navigation (VLN) is a challenging task due to its large searching space in the environment. To address this problem, previous works have proposed some methods of fine-tuning a large model that pretrained on large-scale datasets. However, the conventional fine-tuning methods require extra human-labeled navigation data and lack self-exploration capabilities in environments, which hinders their generalization of unseen scenes. To improve the ability of fast cross-domain adaptation, we propose Pro mptb ased E nvironmental S elf-exploration (ProbES), which can self-explore the environments by sampling trajectories and automatically generates structured instructions via a large-scale cross-modal pretrained model (CLIP). Our method fully utilizes the knowledge learned from CLIP to build an in-domain dataset by self-exploration without human labeling. Unlike the conventional approach of fine-tuning, we introduce prompt-based learning to achieve fast adaptation for language embeddings, which substantially improves the learning efficiency by leveraging prior knowledge. By automatically synthesizing trajectory-instruction pairs in any environment without human supervision and efficient prompt-based learning, our model can adapt to diverse vision-language navigation tasks, including VLN and REVERIE. Both qualitative and quantitative results show that our ProbES significantly improves the generalization ability of the navigation model * . Teaching a robot to navigate following a natural language instruction has a broad impact in the field of human-robotic interaction. Many related tasks have been proposed to delve into this problem. The  Corresponding author. vision-language navigation (VLN) task (Anderson et al., 2018) is proposed where an agent is required to navigate in a photo-realistic environment step-by-step following a natural language instruction. Recent tasks (Qi et al., 2020; Zhu et al., 2021) focus on target objects localization that asks an agent to identify an object in an unseen room. Solving these tasks requires an agent to obtain a vision-text alignment ability that locates related objects and executes corrective actions according to the instruction. However, collecting a large-scale VLN dataset is difficult and laborious since annotating the semantic of a trajectory within a sentence costs times of labor than annotating an image. Existing navigation datasets are relatively small-scale, and learning on such datasets hinders the agent to obtain a good generalization ability. To solve this problem, EnvDrop (Tan et al., 2019) uses a speaker model to generate instructions for sampled trajectories in unseen environments, but the generalization ability is not strong with limited vision-language understanding ability. Recently, VLN-BERT (Ma-jumdar et al., 2020) introduces a visio-linguistic model pretrained on Conceptual Captions (Sharma et al., 2018) dataset to learn from image-caption pairs, which are quite different from trajectory-instruction pairs from VLN. To address this, Airbert (Guhur et al., 2021) constructs a large-scale in-domain pretraining dataset with image-caption pairs collected from online marketplaces such as Airbnb to finetune ViLBERT. However, Airbert collects image captioning data on websites, which are still far from the scenario of vision-language navigation. Different from previous methods that collect human-labeled data to train a navigation model, we suggest that automatically generating instruction-trajectory pairs by self-exploration for pretraining not only helps the model obtain better generalization ability but also achieves fast adaptation to downstream tasks. prompt-based environmental self-exploration (ProbES) that generates navigation data with prior knowledge automatically and adapts pretrained model quickly to VLN tasks. An overview of our proposed framework is shown in Figure 1. By using this method, a pretrained visio-linguistic model is able to adapt to the VLN task automatically and efficiently. Specifically, we build an in-domain dataset by self-exploration without labeling or crawler. To build such a dataset. we first generate templates by masking visual and action words in labeled instructions. Then, we sample trajectories in the training environment. A pretrained CLIP (Radford et al., 2021) model is used to recognize rooms and objects in the sampled trajectories and match described phrases with them. We construct instructions by filling the matched phrases into sampled templates. By leveraging the prior knowledge learned by CLIP, we are able to build a dataset automatically with rich semantic information. Meanwhile, finetuning the whole pretrained model is time-consuming, we adopt prompt tuning (Li and Liang, 2021; Liu et al., 2021c,b), a lightweight alternative to finetuning. Our prompt-based method can distill task-relevant knowledge from pretrained model and achieve fast adaption to downstream tasks. We evaluate ProbES on R2R (Anderson et al., 2018) and REVERIE (Qi et al., 2020) datasets by discriminative and generative settings. Results show that ProbES can match or surpass the performance of finetuning with substantially less training time. (1) We propose ProbES, a novel self-exploration method to automatically build an in-domain dataset that reduces the domain gap between the pretraining dataset and VLN tasks without human labeling; (2) Compared with finetuning large pretrained model, our proposed prompt tuning can achieve fast adaptation; (3) Experiments are conducted on R2R and REVERIE datasets with generative and discriminative settings, and results indicate that our proposed ProbES can achieve better or comparable performance. Besides, our generated data can be used as augmented data which improves the generalization ability of the model. Vision-and-Language Navigation. Anderson et al. (Anderson et al., 2018) proposed the first Vision-Language Navigation (VLN) benchmark combining real imagery (Chang et al., 2017) and natural language navigation instructions. To solve this task, Wang et al. (Wang et al., 2020) proposed a novel SERL model to learn reward functions from the expert distribution. And combining imitation learning and reinforcement learning (Wang et al., 2019) has been proved to be beneficial for VLN. Since the VLN dataset is relatively small-scale, some works propose augmentation approaches (Fried et al., 2018; Tan et al., 2019; Liu et al., 2021a) to improve robustness. Auxiliary losses (Majumdar et al., 2020; Zhu et al., 2020; Liang et al., 2021) is used to take advantage of the additional training signals derived from the semantic information. Some pretraining methods (Huang et al., 2019; Hao et al., 4838 2020) have been proposed to learn generic cross-modal representations. This is further extended to a recurrent model that significantly improves sequential action prediction (Hong et al., 2021). However, the limited number of environments in pretraining constrain the generalization ability to unseen scenarios. Most related to this work, VLN-BERT (Majumdar et al., 2020) transfers knowledge from abundant, but out-of-domain image-text data to improve path-instruction matching. In contrast, we not only propose an effective method to build an in-domain dataset by sampling trajectory and generating instructions with templates, but also present a prompt-based pretraining strategy to improve VLN. Vision-and-Language Pretraining. Vision-and-language pretraining has made great progress in recent years. Inspired by BERT (Devlin et al., 2019), much work has extended it to process visual tokens and pretrain on large-scale image-text pairs for learning generic visio-linguistic representations. Previous research introduces one-stream BERT models and two-stream BERT models. The former directly perform inter-modal grounding (Li et al., 2019; Su et al., 2019; Alberti et al., 2019; Li et al., 2020a; Chen et al., 2020; Zhou et al., 2020; Li et al., 2020b), while two-stream models process both visual and textual inputs in separate streams, and then fuse the two modalities in a later stage (Lu et al., 2019; Tan and Bansal, 2019). These models are often pretrained with self-supervised objectives akin to those in BERT: masked language modeling, masked object classification, and sentence-image alignment. In this work, the architecture of the ProbES model is structural similar to ViLBERT (Lu et al., 2019). We make several VLN-specific adaptations to ViLBERT so that pretrained weights can be transferred to initialize large portions of the model. Different from VLN-BERT which fine-tunes a ViLBERT on instruction-trajectory pairs to measure their compatibility in beam search setting, we introduce prompt tuning, which only tunes the continuous prompts. Prompting. Natural language prompting freezes pretrained models and reformats the natural language input with example prompts. GPT-3 (Brown et al., 2020) introduces in-context learning, using manually designed and discrete text prompts. Sun et al. (Sun and Lai, 2020) also leverage prompts as keywords to control the sentiment or topic of the generated sentence. AutoPrompt (Shin et al., 2020) searches for a sequence of discrete trigger words and concatenates it with each input to elicit sentiment or factual knowledge from a masked LM. Different from the discrete text prompt, some methods examine continuous prompts (a.k.a. soft prompts) that perform prompting directly in the embedding space of the model. Prefix-Tuning (Li and Liang, 2021) prepends a sequence of continuous task-specific vectors as virtual tokens to the input. (Zhong et al., 2021; Qin and Eisner, 2021; Hambardzumyan et al., 2021) introduce continuous templates following manual prompt templates. P-tuning (Liu et al., 2021c) uses continuous prompts which are learned by inserting trainable variables into the embedded input. Ptr (Han et al., 2021) adopts manually crafted sub-templates and generates complete templates by logic rules. In ProbES, we prepend continuous task-specific vectors to the embedding of the input instruction and directly tune the embeddings of these vectors. After prompt tuning, the model can be adapted to VLN and REVERIE tasks. The Vision-and-Language Navigation (VLN) task gives a global natural sentence I = { w 0 , ..., w l } as an instruction, where w i is a word token while the l is the length of the sentence. The instruction consists of step-by-step guidance toward the goal. At step t , the agent observes a panoramic view O t = { o t,i } 36 i =1 as the vision input, which is composed of 36 RGB image views. Each of these views consists of image feature v i and an orientation description ( sin  t,i , cos  t,i , sin  t,i , cos  t,i ). Candidates in the panoramic action space consist of k neighbours of the current node in the navigation graph and a stop action. We first generate templates from instructions in the R2R dataset. Then we sample trajectories in the training environment. We generate the candidate noun phrases and actionable verbs for the sampled trajectories and full-fill the templates by the above words. A detailed demonstration of our instruction generation module is shown in Fig. 2. Generating Templates We collect phrases and replace these phrases in human-annotated navigation instruction with blank masks to generate templates. Different from the Airbert (Guhur et al., 2021) that 4839 Env CLIP forward aisle Full-filling FillingModule Turn left and walk through the living room. A trajectory is denoted as { v 1 , v 2 , ..., v n } , where v i represents an observation viewpoint. We introduce CLIP (Radford et al., 2021) to select candidate phrases c and match them to each view v i . We first embed the sentence a photo of [ c noun ]' by CLIP, where the c noun represents the noun-phrase candidates (room or object classes labeled in Matterport dataset). Then we embed the view image by the vision encoder of CLIP and calculate the similarity of the language embedding and vision embedding. We select the candidate with the highest matching score for the view v i . Each view has two matched candidates, one for the detected room and another for an object. Then the description c i of this view is written in 3 formats randomly: [room]', [object]' or [room] with [object]'. Since trajectories are sampled in the environment, we can obtain actionable verbs a i between two viewpoints via comparing headings and elevations. only extracts noun phrases, we also mask action words like left', right', 'forward', and around'. We denote the O mask as the mask for an object and A mask is the mask for an action. The generated templates are like Turn A mask and walk past O mask . Once out, walk A mask O mask . Stop once you reach O mask '. More examples are shown in Table 1. Sampling Trajectories and Actions We first sample the trajectories in the Matterport (Chang et al., 2017) Environment. We randomly sample the starting and ending positions, and collect tracks with lengths of less than 8 hops. Then we obtain the corresponding actions of each trajectory by first-person movement. If the agent chooses the front navigable position to move, we generate a forward' action. If the agent chooses the back navigable position to move, we generate an around' action. Otherwise, if the agent selects the right front navigable position to move for the next step, we generate an action sequence like {right', forward'}, which is used to fill actionable verbs during instruction generation. Full-filling Template with Prior Knowledge Prior knowledge is the key to generating high-quality data without human labeling. ProbES introduces CLIP, a powerful vision-language alignment model learned from a large-scale image-caption dataset. To generate structured augmentation data, we full-fill the templates with phrases that describe the sampled trajectory and actions. We randomly select a template with the same or a close number of O mask as the number of viewpoints in the sampled trajectory. The template has a sequence of object masks { O mask, 1 , O mask, 2 , ..., O mask,i } and a sequence of action masks { A mask, 1 , A mask, 2 , ..., A mask,j } . Lengths of object masks and action masks are denoted as l and n respectively. The number of object masks and action masks is roughly balanced. Let n v be the number of viewpoints in a sampled trajectory. Then the generated captions of this trajectory is written as { c 1 , c 2 , ..., c n v } . We 4840 Table 1: Examples of generated templates. full-fill the templates by the following rules: 1) if n v  l , we randomly sample l captions and fill the O mask in the template sequentially; 2) if n v < l , we randomly sample the O mask and use all the caption phrases to fill them. After filling phrases, we can identify which viewpoint A mask,i may appear since viewpoints of O mask,j near it are already known. For example, if the template is like  O mask, 1 A mask, 1 O mask, 2 ' and captions of v 1 and v 2 are used to fill O mask, 1 and O mask, 2 respectively, then A mask, 1 is the sampled action between v 1 and v 2 . In this way, we use generated actionable verbs to full-fill the templates and get final instructions. By the above method, we can generate diverse instructions without human labeling. Prompt tuning has been found effective on many natural language understanding (NLU) tasks. Motivated by this, we introduce a prompt-based architecture to achieve fast adaptation on the self-exploration dataset (e.g., Conceptual Captions) and downstream tasks. The architecture is ViLBERT-like and equipped with a prompt encoder for prompt tuning. Given an instruction-trajectory pair, the visual and textual features can be extracted by the visual encoder E v and textual encoder E x in ViLBERT respectively. Especially, the textual input has two parts: prompt sequence { p 1 , ..., p n } and word sequence { x 1 , ..., x m } , where p and x indicate a pseudo prompt token and a word token of a generated instruction respectively. n and m represent lengths of the prompt sequence and word sequence respectively. We embed prompt sequence by the prompt encoder E p and embed word sequence by the textual encoder E x as follows: e p, 1 , ..., e p,n = E p ( p 1 , ..., p n ) e x, 1 , ..., e x,m = E x ( x 1 ) , ..., E x ( x m ) , (1) where E p is composed of a LSTM head followed by a MLP head. Then the textual embedding is mapped to e t = { e p, 1 , ..., e p,n , e x, 1 , ..., e x,m } , where e p, 1 , ..., e p,n are trainable embedding tensors and enable us to find better continous prompts. Let e v be denoted as visual embedding produced by visual encoder E v . e t and e v are then passed to the co-attention transformer similar to ViLBERT. Then in the prompt tuning process, we only train E p and fix the parameters of E x for the language stream. For the vision stream, since the trajectory is represented as a sequence of panoramic image regions, which is different from VLMs pretrained on image-caption pairs, we also update the visual embedding during prompt tuning. The visual embedding contains image embedding and location embedding. We sample hard negative paths based on distance in the environment for an instruction-trajectory pair, and the model is trained to choose the best path among them. Our model can adapt to diverse downstream navigation tasks, including VLN, a step-by-step navigation task, and REVERIE, an object-oriented navigation task. In the step-by-step navigation task, our model receives an instruction sentence and navigates following the commands in the instruction sequentially. In the object navigation task, our model receives an object description and explores the house to find an object. Also, our model can be adapted to both discriminative and generative navigation settings. In the discriminative setting, our model receives both an instruction and the observation sequence to represent a navigation trajectory and then output a score. In the generative setting, our model receives instruction and predicts actions sequentially. We experiment with our proposed ProbES on two downstream tasks: goal-oriented navigation task (R2R (Anderson et al., 2018)), and object-oriented navigation task (REVERIE (Qi et al., 2020)). ProbES can be easily applied to discriminative and generative models for these two tasks. Evaluation Metrics A large number of metrics are used to evaluate models in VLN, such as Trajectory Length (TL), the trajectory length in meters, Navigation Error (NE), the navigation error in meters, Oracle Success Rate (OR), the rate if the agent successfully stops at the closest point, Success Rate (SR), the success rate of reaching the goal, and Success rate weighted by (normalized inverse) Path Length (SPL) (Anderson et al., 2018). VLN task regard SR and SPL as the primary metric, and the REVERIE task regard RGS and RGSPL as the primary metric. Implementation Details Our training process is divided into two steps: Firstly, we pretrain our model on our generated self-exploration training set with prompt tuning for only 10 epochs. After that, we adapt our model to the downstream discriminative VLN task with only ranking loss for 20 epochs. The batch size is set as 64 and the learn-Table 4: Results by comparing ProbES with VLN-BERT in discriminative setting. ing rate is 4  10  5 . The generative navigation settings are the same as Recurrent VLN-BERT on both R2R and REVERIE. During pretraining, we use ProbES to 50k instruction-trajectory pairs. We use 32 NVIDIA V100 GPUs for pretraining and 8 GPUs for adaptation. Experiments with generative settings are conducted on a V100 GPU. In this section, we compare our model with previous state-of-the-art methods. We compare the ProbES with two baselines (ViLBERT and VLN-BERT built on Recurrent VLN-Bert) and five other methods. A brief description of previous models is as followed: 1) Seq2Seq: A sequence to sequence model reported in (Anderson et al., 2018); 2) Speaker-Follower (Fried et al., 2018): a method introduces a data augmentation approach and panoramic action space; 3) PRESS (Li et al., 2019): a conventional fine-tuning method with stochastic instruction sampling; 4) EnvDrop (Tan 4842 et al., 2019): a method augment data with environmental dropout; 5) Recurrent VLN-Bert (Hong et al., 2021) on three different settings: OSCAR and ViLBERT pretrained on out-of-domain data, VLN-BERT pretrained on R2R. We compare the models on three splits in the R2R dataset: validation seen house, validation unseen house, and testing (where the houses are also unseen). We also compare ProbES with Seq2Seq, RCM (Wang et al., 2019), SMNA (Ma et al., 2019), FAST-MATTN (Qi et al., 2020), Recurrent VLN-Bert (Hong et al., 2021) on OSCAR on REVERIE dataset. Results on R2R We compare ProbES with previous state-of-the-art methods on the R2R dataset in the generative setting, which predicts actions sequentially, as shown in Table 2. Rec indicates using Recurrent VLN-Bert (Hong et al., 2021) with different backbones or parameter initialization. In the validation seen split, compared to VLN-BERT under the same setting, our ProbES achieves 5% improvement on SR and 5% improvement on SPL. In the validation unseen split, we achieve 1% improvement on SR compared to VLN-BERT. In the testing split, ProbES shows competitive results. Note that the PREVALENT backbone is pretrained on an in-domain R2R dataset with scene features and fine-tuned with an additional action prediction task in a generative setting while ProbES does not use labeled R2R data or augmented data generated by speaker (Fried et al., 2018). Results in Discriminative Setting We compare ProbES with VLN-BERT in the discriminative setting, which outputs scores for instruction-trajectory pairs, as in Table 4. In the validation unseen split, our method outperforms VLN-BERT, which indicates ProbES is able to improve the generalization ability for unseen scenes. Results on REVERIE We compare ProbES with previous state-of-the-art methods on the REVERIE dataset, as shown in Table 3. In the validation unseen split, we achieve 0.42% improvement on RGS and 0.65% improvement on RGSPL. In the testing split, ProbES achieves 0.87% improvement on RGS and 0.69% improvement on RGSPL. We can see that ProbES benefits from prompt tuning with our generated instruction-trajectory pairs. Ablation of Learning Strategies. In Table 5, we ablate the performance gains from different learning strategies. PT and FT represent prompt tun-Table 5: Ablation of different modules during pretraining and finetuning. ing and fine-tuning respectively. Mask and Rank stand for masked multi-modal modeling loss and the ranking loss for path-selection task. We regard the model finetuned by ranking loss as our baseline. The masked multi-modal modeling loss on our data and R2R data are able to improve the performance. And finetuning on our data is able to improve generalization ability since the success rate in the validation unseen split gets 1.1% improvement and achieves 59.0%. At last, we discover that pretraining on our data with prompt tuning improves the baseline performance by 20.8% in the validation unseen split, achieving the best performance. Our model outperforms the model fine-tuned on R2R dataset by 1.1% in unseen split, indicating that ProbES improves the generalization ability of the navigation model. Ablation of Instruction Generation. Table 6 introduces comprehensive ablation experiments showing the impact of key steps in the strategy of generating instructions, and the experiments are performed in the baseline model: IL + RL from EnvDrop (Tan et al., 2019). Class indicates classes we use to feed into CLIP. M and P/O represent classes from Matterport and Place365/Objects365 datasets respectively. G Template denotes the strategy used to generate templates. ours' denote the strategy shown in Sec 3.2. For S Template , random' and match' indicate sampling a template randomly and choosing a template with the same number of masks as the number of viewpoints. As shown in Table 6, randomly selecting template without considering the number of masked tokens degrades the performance and introduces more noise in the data. Results show that equipped with our generated data (Row 3) improves the performance by a large margin. The model of using the rooms and objects from Places365 (Zhou et al., 2017) and Objects365 (Shao et al., 2019) (Row 4) performs worse than which uses the rooms and objects from Matterport. We infer from that Places365 and Objects365 contain many outdoor 4843 Figure 3: Statistical analysis of generated instructions. scenes and objects which are not suitable for VLN. Visualization of Data Distribution Figure 3 presents a statistical analysis of our generated instructions. We can see from the left figure that the number of object masks are larger than that of action masks, indicating that instructions contain more rich information generated by CLIP from sampled observations. The right figure shows the distribution of the instruction lengths. The lengths of most of the instructions range from 10 to 30, which matches the R2R dataset. The easy samples and hard samples in our generated instructions are balanced. Visualization of Trajectory-instruction pairs Here we provide visualization of the data generated by ProbES. Figure 4 shows the instruction-trajectory samples generated with our strategy. For each sample, we visualize observations of the trajectory, captions generated with CLIP, the selected template, and the final instruction generated by ProbES. Generated object classes fit observed 4844 scenes well, thus we can infer that CLIP is able to extract key information from the observation. Also, our method can select a suitable template and generate diverse instructions that describe observations of trajectories correctly. The length of our generated instruction ranges from 1 to 3 sentences, which matches the data distribution of the R2R dataset. In this work, we first introduce an effective way to generate in-domain data for pretraining the VLN model: leveraging a large pretrained CLIP model to generate captions for each viewpoint and sampling actions in the environment. Experiments show that the domain gap between pretraining data and VLN tasks can be mitigated. We also propose a prompt-based architecture, which introduces prompt tuning to adapt the pretrained model fastly. Our proposed ProbES achieves better results compared to baseline on both R2R and REVERIE datasets, and ablations show the contribution of each module and the effectiveness of the generated data. This work was supported in part by National Natural Science Foundation of China (NSFC) No.61976233, Guangdong Province Basic and Applied Basic Research (Regional Joint Fund-Key) Grant No.2019B1515120039, Guangdong Outstanding Youth Fund (Grant No. 2021B1515020061), Shenzhen Fundamental Research Program (Project No. RCYX20200714114642083, No. JCYJ20190807154211365) and CAAI-Huawei MindSpore Open Fund. We thank MindSpore for the partial support of this work, which is a new deep learning computing framwork  , and supported by Guangdong Provincial Key Laboratory of Fire Science and Intelligent Emergency Technology, Guangzhou 510006, China.
abstain abstain abstain objective method result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain objective abstain method abstain objective method abstain method method method result method abstain other objective other other other other other other other other other abstain objective other other other other other other abstain method method other other other other other other other other other other abstain other abstain abstain abstain abstain abstain objective method method result method abstain abstain abstain objective method abstain abstain abstain result method abstain abstain abstain abstain method objective method method method abstain abstain method abstain abstain abstain abstain abstain abstain method method method abstain method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain method abstain method abstain abstain abstain abstain method abstain other abstain method method abstain abstain abstain method method abstain method method other result method result abstain result result abstain abstain result result result result abstain result abstain abstain abstain abstain abstain abstain result result result result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain result abstain result abstain abstain objective abstain objective objective other other