summaries: This submission introduces a new method for vision+language navigation which tracks progress on the instruction using a progress monitor and a visual-textual co-grounding module. The method is shown to perform well on a standard benchmark. Ablation tests indicate the importance of each component of the model. Qualitative examples show that the proposed method attends to different parts of the instruction as the agent moves.  The paper considers the problem of following natural language route instructions in an unknown environment given only images. Integral to the proposed ("self-aware") approach is its ability to reason over which aspects of the instruction have been completed, which are to be followed next, which direction to go in next, as well as the agents current progress. This involves two primary components of the architecture. The first is a visual-textual module that grounds to the completed instruction, the next instruction, and the next direction based upon the visual input. The second is a "progress monitor" that takes the grounded instruction as input and captures the agent's progress towards completing the instruction.
abstractText: The Vision-and-Language Navigation (VLN) task entails an agent following navigational instruction in photo-realistic unknown environments. This challenging task demands that the agent be aware of which instruction was completed, which instruction is needed next, which way to go, and its navigation progress towards the goal. In this paper, we introduce a self-monitoring agent with two complementary components: (1) visual-textual co-grounding module to locate the instruction completed in the past, the instruction required for the next action, and the next moving direction from surrounding images and (2) progress monitor to ensure the grounded instruction correctly reflects the navigation progress. We test our selfmonitoring agent on a standard benchmark and analyze our proposed approach through a series of ablation studies that elucidate the contributions of the primary components. Using our proposed method, we set the new state of the art by a significant margin (8% absolute increase in success rate on the unseen test set). Code is available at https://github.com/chihyaoma/selfmonitoring-agent.
authors.name: Chih-Yao Ma
authors.name: Jiasen Lu
authors.name: Zuxuan Wu
authors.name: Ghassan AlRegib
authors.name: Zsolt Kira
authors.name: Richard Socher
authors.name: Caiming Xiong
references.authors: Aishwarya Agrawal Dhruv Batra Devi Parikh Aniruddha Kembhavi
references.title: Dont just assume; look and answer: Overcoming priors for visual question answering
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Muhannad Al-Omari Paul Duckworth David C Hogg Anthony G Cohn
references.title: Natural language acquisition and grounding for embodied robotic systems
references.venue: In Association for the Advancement of Artificial Intelligence (AAAI),
references.authors: Peter Anderson Angel Chang Devendra Singh Chaplot Alexey Dosovitskiy Saurabh Gupta Vladlen Koltun Jana Kosecka Jitendra Malik Roozbeh Mottaghi Manolis Savva
references.title: On evaluation of embodied navigation agents
references.venue: arXiv preprint arXiv:1807.06757,
references.authors: Peter Anderson Qi Wu Damien Teney Jake Bruce Mark Johnson Niko Sünderhauf Ian Reid Stephen Gould Anton van den Hengel
references.title: Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Jacob Andreas Dan Klein
references.title: Alignment-based compositional semantics for instruction following
references.venue: In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP),
references.authors: Stanislaw Antol Aishwarya Agrawal Jiasen Lu Margaret Mitchell Dhruv Batra C Lawrence Zitnick Devi Parikh
references.title: Vqa: Visual question answering
references.venue: Proceedings of the IEEE International Conference on Computer Vision (ICCV),
references.authors: Jacob Arkin Matthew R Walter Adrian Boteanu Michael E Napoli Harel Biggie Hadas KressGazit Thomas M Howard
references.title: Contextual awareness: Understanding monologic natural language instructions for autonomous robots
references.venue: In Robot and Human Interactive Communication (RO-MAN),
references.authors: Yoav Artzi Luke Zettlemoyer
references.title: Weakly supervised learning of semantic parsers for mapping instructions to actions
references.venue: Transactions of the Association of Computational Linguistics (ACL),
references.authors: Yael Benn Thomas L Webb Betty PI Chang Yu-Hsuan Sun Iain D Wilkinson Tom FD Farrow
references.title: The neural basis of monitoring goal progress
references.venue: Frontiers in human neuroscience,
references.authors: Elliot T Berkman Matthew D Lieberman
references.title: The neuroscience of goal pursuit
references.venue: The psychology of goals,
references.authors: Satchuthananthavale RK Branavan Harr Chen Luke S Zettlemoyer Regina Barzilay
references.title: Reinforcement learning for mapping instructions to actions
references.venue: In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-Volume
references.authors: Angel Chang Angela Dai Thomas Funkhouser Maciej Halber Matthias Niessner Manolis Savva Shuran Song Andy Zeng Yinda Zhang
references.title: Matterport3D: Learning from RGB-D data in indoor environments
references.venue: International Conference on 3D Vision
references.authors: Christopher H Chatham Eric D Claus Albert Kim Tim Curran Marie T Banich Yuko Munakata
references.title: Cognitive control reflects context monitoring, not motoric stopping, in response inhibition
references.venue: PloS one,
references.authors: Trevor Cohn Cong Duy Vu Hoang Ekaterina Vymolova Kaisheng Yao Chris Dyer Gholamreza Haffari
references.title: Incorporating structural alignment biases into an attentional neural translation model
references.venue: In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL),
references.authors: Abhishek Das Satwik Kottur Khushi Gupta Avi Singh Deshraj Yadav José M.F. Moura Devi Parikh Dhruv Batra
references.title: Visual Dialog
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Abhishek Das Samyak Datta Georgia Gkioxari Stefan Lee Devi Parikh Dhruv Batra
references.title: Embodied question answering
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
references.authors: Harm de Vries Kurt Shuster Dhruv Batra Devi Parikh Jason Weston Douwe Kiela
references.title: Talk the walk: Navigating new york city through grounded dialogue
references.venue: arXiv preprint arXiv:1807.03367,
references.authors: Krishna SR Dubba Miguel R De Oliveira Gi Hyun Lim Hamidreza Kasaei L Seabra Lopes Ana Tomé Anthony G Cohn
references.title: Grounding language in perception for scene conceptualization in autonomous robots. In Qualitative Representations for Robots: Papers from the AAAI Spring Symposium
references.venue: Technical report,
references.authors: Felix Duvallet Matthew R Walter Thomas Howard Sachithra Hemachandra Jean Oh Seth Teller Nicholas Roy Anthony Stentz
references.title: Inferring maps and behaviors from natural language instructions
references.venue: In Experimental Robotics,
references.authors: Daniel Fried Ronghang Hu Volkan Cirik Anna Rohrbach Jacob Andreas Louis-Philippe Morency Taylor Berg-Kirkpatrick Kate Saenko Dan Klein Trevor Darrell
references.title: Speaker-follower models for vision-and-language navigation
references.venue: In Advances in Neural Information Processing Systems (NIPS),
references.authors: Kirill Gavrilyuk Amir Ghodrati Zhenyang Li Cees GM Snoek
references.title: Actor and action video segmentation from a sentence
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Daniel Gordon Aniruddha Kembhavi Mohammad Rastegari Joseph Redmon Dieter Fox Ali Farhadi
references.title: Iqa: Visual question answering in interactive environments
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Sachithra Hemachandra Felix Duvallet Thomas M. Howard Nicholas Roy Anthony Stentz Matthew R. Walter
references.title: Learning models for following natural language directions in unknown environments
references.venue: In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA),
references.authors: Ronghang Hu Huazhe Xu Marcus Rohrbach Jiashi Feng Kate Saenko Trevor Darrell
references.title: Natural language object retrieval
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Michał Kempka Marek Wydmuch Grzegorz Runc Jakub Toczek Wojciech Jaśkowski
references.title: Vizdoom: A doom-based ai research platform for visual reinforcement learning
references.venue: Proceedings of the IEEE Conference on Computational Intelligence and Games (CIG),
references.authors: Thomas Kollar Jayant Krishnamurthy Grant P Strimel
references.title: Toward interactive grounded language acqusition
references.venue: In Robotics: Science and systems,
references.authors: Cewu Lu Ranjay Krishna Michael Bernstein Li Fei-Fei
references.title: Visual relationship detection with language priors
references.venue: In European Conference on Computer Vision (ECCV),
references.authors: Jiasen Lu Jianwei Yang Dhruv Batra Devi Parikh
references.title: Hierarchical question-image co-attention for visual question answering
references.venue: In Advances In Neural Information Processing Systems,
references.authors: Jiasen Lu Jianwei Yang Dhruv Batra Devi Parikh
references.title: Neural baby talk
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Chih-Yao Ma Asim Kadav Iain Melvin Zsolt Kira Ghassan AlRegib Hans Peter Graf
references.title: Attend and interact: Higher-order object interactions for video understanding
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Matt MacMahon Brian Stankiewicz Benjamin Kuipers
references.title: Walk the talk: Connecting language, knowledge, and action in route instructions
references.venue: In Association for the Advancement of Artificial Intelligence (AAAI),
references.authors: Cynthia Matuszek Evan Herbst Luke Zettlemoyer Dieter Fox
references.title: Learning to parse natural language commands to a robot control system
references.venue: In Experimental Robotics,
references.authors: Hongyuan Mei Mohit Bansal Matthew R Walter
references.title: Listen, attend, and walk: Neural mapping of navigational instructions to action sequences
references.venue: In Association for the Advancement of Artificial Intelligence (AAAI),
references.authors: Piotr Mirowski Razvan Pascanu Fabio Viola Hubert Soyer Andrew J Ballard Andrea Banino Misha Denil Ross Goroshin Laurent Sifre Koray Kavukcuoglu
references.title: Learning to navigate in complex environments
references.venue: In Proceedings of the International Conference on Learning Representations (ICLR),
references.authors: Piotr Mirowski Matthew Koichi Grimes Mateusz Malinowski Karl Moritz Hermann Keith Anderson Denis Teplyashin Karen Simonyan Koray Kavukcuoglu Andrew Zisserman Raia Hadsell
references.title: Learning to navigate in cities without a map
references.venue: arXiv preprint arXiv:1804.00168,
references.authors: Dipendra Misra John Langford Yoav Artzi
references.title: Mapping instructions and visual observations to actions with reinforcement learning
references.venue: In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP),
references.authors: Dipendra K Misra Jaeyong Sung Kevin Lee Ashutosh Saxena
references.title: Tell me dave: Context-sensitive grounding of natural language to manipulation instructions
references.venue: The International Journal of Robotics Research,
references.authors: Arsalan Mousavian Alexander Toshev Marek Fiser Jana Kosecka James Davidson
references.title: Visual representations for semantic target driven navigation
references.venue: arXiv preprint arXiv:1805.06066,
references.authors: Varun K Nagaraja Vlad I Morariu Larry S Davis
references.title: Modeling context between objects for referring expression understanding
references.venue: In European Conference on Computer Vision (ECCV),
references.authors: Anna Rohrbach Marcus Rohrbach Ronghang Hu Trevor Darrell Bernt Schiele
references.title: Grounding of textual phrases in images by reconstruction
references.venue: In European Conference on Computer Vision (ECCV),
references.authors: Michael Spranger Luc Steels
references.title: Co-acquisition of syntax and semantics-an investigation in spatial language
references.venue: In Association for the Advancement of Artificial Intelligence (AAAI),
references.authors: Makarand Tapaswi Yukun Zhu Rainer Stiefelhagen Antonio Torralba Raquel Urtasun Sanja Fidler
references.title: Movieqa: Understanding stories in movies through question-answering
references.venue: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Stefanie Tellex Thomas Kollar Steven Dickerson Matthew R Walter Ashis Gopal Banerjee Seth J Teller Nicholas Roy
references.title: Understanding natural language commands for robotic navigation and mobile manipulation
references.venue: In Association for the Advancement of Artificial Intelligence (AAAI),
references.authors: Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Łukasz Kaiser Illia Polosukhin
references.title: Attention is all you need
references.venue: In Advances in Neural Information Processing Systems (NIPS),
references.authors: Adam Vogel Dan Jurafsky
references.title: Learning to follow navigational directions
references.venue: In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL),
references.authors: Tsun-Hsuan Wang Hung-Jui Huang Juan-Ting Lin Chan-Wei Hu Kuo-Hao Zeng Min Sun
references.title: Omnidirectional cnn for visual place recognition and navigation
references.venue: Proceedings of the IEEE Conference on Robotics and Automation (ICRA),
references.authors: Xin Wang Wenhan Xiong Hongmin Wang William Yang Wang
references.title: Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation
references.venue: In European Conference on Computer Vision (ECCV),
references.authors: Greg Wayne Chia-Chun Hung David Amos Mehdi Mirza Arun Ahuja Agnieszka GrabskaBarwinska Jack Rae Piotr Mirowski Joel Z Leibo Adam Santoro
references.title: Unsupervised predictive memory in a goal-directed agent
references.venue: arXiv preprint arXiv:1803.10760,
references.authors: Haonan Yu Xiaochen Lian Haichao Zhang Wei Xu
references.title: Guided feature transformation (gft): A neural language grounding module for embodied agents
references.venue: arXiv preprint arXiv:1805.08329,
references.authors: Daniel Gordon
references.title: Visual Semantic Planning using Deep Successor Representations
references.venue: Proceedings of the IEEE International Conference on Computer Vision (ICCV),
references.authors: AR Zamir F Xia J He A Sax J Malik S Savarese. Gibson
references.title: env: Real-world perception for embodied agents
references.venue: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Luowei Zhou Chenliang Xu Jason J Corso
references.title: Towards automatic learning of procedures from web instructional videos
references.venue: In Association for the Advancement of Artificial Intelligence (AAAI),
references.authors: Luowei Zhou Yingbo Zhou Jason J Corso Richard Socher Caiming Xiong
references.title: End-to-end dense video captioning with masked transformer
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Yuke Zhu Roozbeh Mottaghi Eric Kolve Joseph J Lim Abhinav Gupta Li Fei-Fei Ali Farhadi
references.title: Target-driven visual navigation in indoor scenes using deep reinforcement learning
references.venue: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA),
references.authors: Fried
references.title: 2018), the 4-dim orientation features are tiled 32 times, resulting a embedding feature
sections.heading: 1 INTRODUCTION
sections.text: Recently, the Vision-and-Language (VLN) navigation task (Anderson et al., 2018b), which requires the agent to follow natural language instructions to navigate through a photo-realistic unknown environment, has received significant attention (Wang et al., 2018b; Fried et al., 2018). In the VLN task, an agent is placed in an unknown realistic environment and is required to follow natural language instructions to navigate from its starting location to a target location. In contrast to some existing navigation tasks (Kempka et al., 2016; Zhu et al., 2017; Mirowski et al., 2017; 2018), we address the class of tasks where the agent does not have an explicit representation of the target (e.g., location in a map or image representation of the goal) to know if the goal has been reached or not (Matuszek et al., 2013; Hemachandra et al., 2015; Duvallet et al., 2016; Arkin et al., 2017). Instead, the agent needs to be aware of its navigation status through the association between the sequence of observed visual inputs to instructions.
  Consider an example as shown in Fig. 1, given the instruction ”Exit the bedroom and go towards the table. Go to the stairs on the left of the couch. Wait on the third step.”, the agent first needs to locate which instruction is needed for the next movement, which in turn requires the agent to be aware of (i.e., to explicitly represent or have an attentional focus on) which instructions were completed or ongoing in the previous steps. For instance, the action ”Go to the stairs” should be carried out once the agent has exited the room and moved towards the table. However, there exists inherent ambiguity for ”go towards the table”. Intuitively, the agent is expected to ”Go to the stairs” after completing ”go towards the table”. But, it is not clear what defines the completeness of ”Go towards the table”. The completeness of an ongoing action often depends on the availability of the next action. Since the transition between past and next part of the instructions is a soft
  ∗Work done while the authors were research interns at Salesforce Research.
  boundary, in order to determine when to transit and to follow the instruction correctly the agent is required to keep track of both grounded instructions. On the other hand, assessing the progress made towards the goal has indeed been shown to be important for goal-directed tasks in humans decision-making (Benn et al., 2014; Chatham et al., 2012; Berkman & Lieberman, 2009). While a number of approaches have been proposed for VLN (Anderson et al., 2018b; Wang et al., 2018b; Fried et al., 2018), previous approaches generally are not aware of which instruction is next nor progress towards the goal; indeed, we qualitatively show that even the attentional mechanism of the baseline does not successfully track this information through time.
  In this paper, we propose an agent endowed with the following abilities: (1) identify which direction to go by finding the part of the instruction that corresponds to the observed images—visual grounding, (2) identify which part of the instruction has been completed or ongoing and which part is potentially needed for the next action selection—textual grounding, and (3) ensure that the grounded instruction can correctly be used to estimate the progress made towards the goal, and apply regularization to ensure this —progress monitoring. Therefore, we introduce the self-monitoring agent consisting of two complementary modules: visual-textual co-grounding and progress monitor.
  More specifically, we achieve both visual and textual grounding simultaneously by incorporating the full history of grounded instruction, observed images, and selected actions into the agent. We leverage the structural bias between the words in instructions used for action selection and progress made towards the goal and propose a new objective function for the agent to measure how well it can estimate the completeness of instruction-following. We then demonstrate that by conditioning on the positions and weights of grounded instruction as input, the agent can be self-monitoring of its progress and further ensure that the textual grounding accurately reflects the progress made.
  Overall, we propose a novel self-monitoring agent for VLN and make the following contributions: (1) We introduce the visual-textual co-grounding module, which performs grounding interdependently across both visual and textual modalities. We show that it can outperform the baseline method by a large margin. (2) We propose to equip the self-monitoring agent with a progress monitor, and for navigation tasks involving instructions instantiate this by introducing a new objective function for training. We demonstrate that, unlike the baseline method, the position of grounded instruction can follow both past and future instructions, thereby tracking progress to the goal. (3) With the proposed self-monitoring agent, we set the new state-of-the-art performance on both seen and unseen environments on the standard benchmark. With 8% absolute improvement in success rate on the unseen test set, we are ranked #1 on the challenge leaderboard.
sections.heading: 2 SELF-MONITORING NAVIGATION AGENT
sections.heading: 2.1 NOTATION
sections.text: Given a natural language instruction with L words, its representation is denoted by X ={ x1,x2, . . . ,xL } , where xl is the feature vector for the l-th word encoded by an LSTM language encoder. Following Fried et al. (2018), we enable the agent with panoramic view. At each time step, the agent perceives a set of images at each viewpoint vt = { vt,1,vt,2, ...,vt,K } , where K
  is the maximum number of navigable directions1, and vt,k represents the image feature of direction k. The co-grounding feature of instruction and image are denoted as x̂t and v̂t respectively. The selected action is denoted as at. The learnable weights are denoted with W , with appropriate sub/super-scripts as necessary. We omit the bias term b to avoid notational clutter in the exposition.
sections.heading: 2.2 VISUAL AND TEXTUAL CO-GROUNDING
sections.text: First, we propose a visual and textual co-grounding model for the vision and language navigation task, as illustrated in Fig. 2. We model the agent with a sequence-to-sequence architecture with attention by using a recurrent neural network. More specifically, we use Long Short Term Memory (LSTM) to carry the flow of information effectively. At each step t, the decoder observes representations of the current attended panoramic image feature v̂t, previous selected action at−1 and current grounded instruction feature x̂t as input, and outputs an encoder context ht:
  ht = LSTM([x̂t, v̂t,at−1]) (1)
  where [, ] denotes concatenation. The previous encoder context ht−1 is used to obtain the textual grounding feature x̂t and visual grounding feature v̂t, whereas we use current encoder context ht to obtain next action at, all of which will be illustrated in the rest of the section.
  Textual grounding. When the agent moves from one viewpoint to another, it is required to identify which direction to go by relying on a grounded instruction, i.e. which parts of the instruction should be used. This can either be the instruction matched with the past (ongoing action) or predicted for the future (next action). To capture the relative position between words within an instruction, we incorporate the positional encoding PE(·) (Vaswani et al., 2017) into the instruction features. We then perform soft-attention on the instruction features X , as shown on the left side of Fig. 2. The attention distribution over L words of the instructions is computed as:
  ztextualt,l = (Wxht−1) >PE(xl), and αt = softmax(ztextualt ), (2)
  where Wx are parameters to be learnt. ztextualt,l is a scalar value computed as the correlation between word l of the instruction and previous hidden state ht−1, andαt is the attention weight over features in X at time t. Based on the textual attention distribution, the grounded textual feature x̂t can be obtained by the weighted sum over the textual features x̂t = αTt X .
  1Empirically, we found that using only the images on navigable directions to be slightly better than using all 36 surrounding images (12 headings × 3 elevations with 30 degree intervals).
  Visual grounding. In order to locate the completed or ongoing instruction, the agent needs to keep track of the sequence of images observed along the navigation trajectory. We thus perform visual attention over the surrounding views based on its previous hidden vector ht−1. The visual attention weight βt can be obtained as:
  zvisualt,k = (Wvht−1) >g(vt,k), and βt = softmax(zvisualt ), (3)
  where g is a one-layer Multi-Layer Perceptron (MLP), Wv are parameters to be learnt. Similar to Eq. 2, the grounded visual feature v̂t can be obtained by the weighted sum over the visual features v̂t = β T t V .
  Action selection. To make a decision on which direction to go, the agent finds the image features on navigable directions with the highest correlation with the grounded navigation instruction x̂t and the current hidden state ht. We use the inner-product to compute the correlation, and the probability of each navigable direction is then computed as:
  ot,k = (Wa[ht, x̂t]) >g(vt,k) and pt = softmax(ot), (4)
  where Wa are the learnt parameters, g(·) is the same MLP as in Eq. 3, and pt is the probability of each navigable direction at time t. We use categorical sampling during training to select the next action at. Unlike the previous method with the panoramic view (Fried et al., 2018), which attends to instructions only based on the history of observed images, we achieve both textual and visual grounding using the shared hidden state output containing grounded information from both textual and visual modalities. During action selection, we rely on both hidden state output and grounded instruction, instead of only relying on grounded instruction.
sections.heading: 2.3 PROGRESS MONITOR
sections.text: It is imperative that the textual-grounding correctly reflects the progress towards the goal, since the agent can then implicitly know where it is now and what the next instruction to be completed will be. In the visual-textual co-grounding module, we ensure that the grounded instruction reasonably informs decision making when selecting a navigable direction. This is necessary but not sufficient for ensuring that the notion of progress to the goal is encoded. Thus, we propose to equip the agent with a progress monitor that serves as regularizer during training and prunes unfinished trajectories during inference.
  Since the positions of localized instruction can be a strong indication of the navigation progress due to the structural alignment bias between navigation steps and instruction, the progress monitor can estimate how close the current viewpoint is to the final goal by conditioning on the positions and weights of grounded instruction. This can further enforce the result of textual-grounding to align with the progress made towards the goal and to ensure the correctness of the textual-grounding.
  The progress monitor aims to estimate the navigation progress by conditioning on three inputs: the history of grounded images and instructions, the current observation of the surrounding images, and the positions of grounded instructions. We therefore represent these inputs by using (1) the previous hidden state ht−1 and the current cell state ct of the LSTM, (2) the grounded surrounding images v̂t, and (3) the distribution of attention weights of textual-grounding αt, as shown at the bottom of Fig. 2 represented by dotted lines.
  Our proposed progress monitor first computes an additional hidden state output hpmt by using grounded image representations v̂t as input, similar to how a regular LSTM computes hidden states except we use concatenation over element-wise addition for empirical reasons2. The hidden state output is then concatenated with the attention weightsαt on textual-grounding to estimate how close the agent is to the goal3. The output of the progress monitor ppmt , which represents the completeness of instruction-following, is computed as:
  hpmt = σ(Wh([ht−1, v̂t])⊗ tanh(ct)), p pm t = tanh(Wpm([αt,h pm t ])) (5)
  where Wh and Wpm are the learnt parameters, ct is the cell state of the LSTM, ⊗ denotes the element-wise product, and σ is the sigmoid function.
  2We found that using concatenation provides slightly better performance and stable training. 3We use zero-padding to handle instructions with various lengths.
sections.heading: 2.4 TRAINING AND INFERENCE
sections.text: Training. We introduce a new objective function to train the proposed progress monitor. The training target ypmt is defined as the normalized distance in units of length from the current viewpoint to the goal, i.e., the target will be 0 at the beginning and closer to 1 as the agent approaches the goal4. Note that the target can also be lower than 0, if the agent’s current distance from the goal is farther than the starting point. Finally, our self-monitoring agent is optimized with a cross-entropy loss and a mean squared error loss, computed with respect to the outputs from both action selection and progress monitor.
  Lloss = −λ T∑
  t=1 ynvt log(pk,t)︸ ︷︷ ︸ action selection
  −(1− λ) T∑
  t=1
  (ypmt − p pm t ) 2
  ︸ ︷︷ ︸ progress monitor
  (6)
  where pk,t is the action probability of each navigable direction, λ = 0.5 is the weight balancing the two losses, and ynvt is the ground-truth navigable direction at step t.
  Inference. During inference, we follow Fried et al. (2018) by using beam search. we propose that, while the agent decides which trajectories in the beams to keep, it is equally important to evaluate the state of the beams on actions as well as on the agent’s confidence in completing the given instruction at each traversed viewpoint. We accomplish this idea by integrating the output of our progress monitor into the accumulated probability of beam search. At each step, when candidate trajectories compete based on accumulated probability, we integrate the estimated completeness of instruction-following ppmt (normalized between 0 to 1) with action probability pk,t to directly evaluate the partial and unfinished candidate routes: pbeamt = p pm t × pk,t.
  Without beam search, we use greedy decoding for action selection with one condition. If the progress monitor output decreases (ppmt+1 < p pm t ), the agent is required to move back to the previous viewpoint and select the action with next highest probability. We repeat this process until the selected action leads to increasing progress monitor output. We denote this procedure as progress inference.
sections.heading: 3 EXPERIMENTS
sections.text: R2R Dataset. We use the Room-to-Room (R2R) dataset (Anderson et al., 2018b) for evaluating our proposed approach. The R2R dataset is built upon the Matterport3D dataset (Chang et al., 2017) and has 7,189 paths sampled from its navigation graphs. Each path has three ground-truth navigation instructions written by humans. The whole dataset is divided into 4 sets: training, validation seen, validation unseen, and test sets unseen.
  Evaluation metrics. We follow the same evaluation metrics used by previous work on the R2R task: (1) Navigation Error (NE), mean of the shortest path distance in meters between the agent’s final
  4We set the target to 1 if the agent’s distance to the goal is less than 3.
  position and the goal location. (2) Success Rate (SR), the percentage of final positions less than 3m away from the goal location. (3) Oracle Success Rate (OSR), the success rate if the agent can stop at the closest point to the goal along its trajectory. In addition, we also include the recently introduced Success rate weighted by (normalized inverse) Path Length (SPL) (Anderson et al., 2018a), which trades-off Success Rate against trajectory length.
sections.heading: 3.1 COMPARISON WITH PRIOR ART
sections.text: We first compare the proposed self-monitoring agent with existing approaches. As shown in Table 1, our method achieves significant performance improvement compared to the state of the arts without data augmentation. We achieve 70% SR on the seen environment and 57% on the unseen environment while the existing best performing method achieved 63% and 50% SR respectively. When trained with synthetic data5, our approach achieves slightly better performance on the seen environments and significantly better performance on both the validation unseen environments and the test unseen environments when submitted to the test server. We achieve 3% and 8% improvement on SR on both validation and test unseen environments. Both results with or without data augmentation indicate that our proposed approach is more generalizable to unseen environments. At the time of writing, our self-monitoring agent is ranked #1 on the challenge leader-board among the state of the arts.
  Note that both Speaker-Follower and our approach in Table 1 use beam search. For comparison without using beam search, please refer to the Appendix.
  Textually grounded agent. Intuitively, an instruction-following agent is required to strongly demonstrate the ability to correctly focus and follow the corresponding part of the instruction as it navigates through an environment. We thus record the distribution of attention weights on instruction at each step as indications of which parts of the instruction being used for action selection. We average all runs across both validation seen and unseen dataset splits. Ideally, we expect to see the distribution of attention weights lies close to a diagonal, where at the beginning, the agent focuses on the beginning of the instruction and shifts its attention towards the end of instruction as it moves closer to the goal.
  To demonstrate, we use the method with panoramic action space proposed in Fried et al. (2018) as a baseline for comparison. As shown in Figure 3, our self-monitoring agent with progress monitor demonstrates that the positions of grounded instruction over time form a line similar to a diagonal. This result may further indicate that the agent successfully utilizes the attention on instruction to complete the task sequentially. We can also see that both agents were able to focus on the first part of the instruction at the beginning of navigation consistently. However, as the agent moves further in unknown environments, our self-monitoring agent can still successfully identify the parts of instruction that are potentially useful for action selection, whereas the baseline approach becomes uncertain about which part of the instruction should be used for selecting an action.
  5We use the exact same synthetic data generated from the Speaker as in Fried et al. (2018) for comparison.
sections.heading: 3.2 ABLATION STUDY
sections.text: We now discuss the importance of each component proposed in this work. We begin with the same baseline as before (agent with panoramic action space in Fried et al. (2018))6.
  Co-grounding. When comparing the baseline with row #1 in our proposed method, we can see that our co-grounding agent outperformed the baseline with a large margin. This is due to the fact that we use the LSTM to carry both the textually and visually grounded content, and the decision on each navigable direction is predicted with both textually grounded instruction and the hidden state output of the LSTM. On the other hand, the baseline agent relies on the LSTM to carry visually grounded content, and uses the hidden state output for predicting the textually grounded instruction. As a result, we observed that instead of predicting the instruction needed for selecting a navigable direction, the textually grounded instruction may match with the past sequence of observed images implicitly saved within the LSTM.
  Progress monitor. Given the effective co-grounding, the proposed progress monitor further ensure that the grounded instruction correctly reflects the progress made toward the goal. This further improves the performance especially on the unseen environments as we can see from row #1 and #2.
  When using the progress inference, the progress monitor serve as a progress indicator for the agent to decide when to move back to the last viewpoint. We can see from row #2 and #4 that the SR performance can be further improved around 2% on both seen and unseen environments.
  Finally, we integrate the output of the progress monitor with the state-factored beam search (Fried et al., 2018), so that the candidate paths compete not only based on the probability of selecting a certain navigable direction but also on the estimated correspondence between the past trajectory and the instruction. As we can see by comparing row #2, #6, and #7, the progress monitor significantly improved the success rate on both seen and unseen environments and is the key for surpassing the state of the arts even without data augmentation. We can also see that when using beam search without progress monitor, the SR on unseen improved 7% (row #1 vs #6), while using beam search integrated with progress estimation improved 13% (row #2 vs #7).
  Data augmentation. In the above, we have shown each row in our approach contributes to the performance. Each of them increases the success rate and reduces the navigation error incrementally. By further combining them with the data augmentation pre-trained from the speaker (Fried et al., 2018), the SR and OSR are further increased, and the NE is also drastically reduced. Interestingly, the performance improvement introduced by data augmentation is smaller than from Speaker-Follower on the validation sets (see Table 1 for comparison). This demonstrates that our proposed method is more data-efficient.
  6Note that our results for this baseline are slightly higher on val-seen and slightly lower on val-unseen than those reported, due to differences in hyper-parameter choices.
sections.heading: 3.3 QUALITATIVE RESULTS
sections.text: To further validate the proposed method, we qualitatively show how the agent navigates through unseen environments by following instructions as shown in Fig. 4. In each figure, the agent follows the grounded instruction (at the top of the figure) and decides to move towards a certain direction (green arrow). For the full figures and more examples of successful and failed agents in both unseen and seen environments, please see the supplementary material.
  Consider the trajectory on the left side in Fig. 4, at step 3, the grounded instruction illustrated that the agent just completed ”turn right” and focuses mainly on ”walk straight to bedroom”. As the agent entered the bedroom, it then shifts the textual grounding to the next action ”Turn left and walk to bed lamp”. Finally, at step 6, the agent completed another ”turn left” and successfully stop at the rug (see the supplementary material for the importance of dealing with duplicate actions). Consider the example on the right side, the agent has already entered the hallway and now turns right to walk across to another room. However, it is ambiguous that which room the instructor is referring to. At step 5, our agent checked out the room on the left first and realized that it does not match with ”Stop in doorway in front of rug”. It then moves to the next room and successfully stops at the goal.
  In both cases, we can see that the completeness estimated by progress monitor gradually increases as the agent steadily navigates toward the goal. We have also observed that the estimated completeness ends up much lower for failure cases (see the supplementary material for further details).
sections.heading: 4 RELATED WORK
sections.text: Vision, Language, and Navigation.. There is a plethora work investigating the combination of vision and language for a multitude of applications (Zhou et al., 2018a;b; Antol et al., 2015; Tapaswi et al., 2016; Das et al., 2017), etc. While success has been achieved in these tasks to handle massive corpora of static visual input and text data, a resurgence of interest focuses on equipping an agent with the ability to interact with its surrounding environment for a particular goal such as object manipulation with instructions (Misra et al., 2016; Arkin et al., 2017), grounded language acquisition (Al-Omari et al., 2017; Kollar et al., 2013; Spranger & Steels, 2015; Dubba et al., 2014),
  embodied question answering (Das et al., 2018; Gordon et al., 2018), and navigation (Matuszek et al., 2013; Hemachandra et al., 2015; Duvallet et al., 2016; Zhu et al., 2017; de Vries et al., 2018; Yuke Zhu, 2017; Mousavian et al., 2018; Wayne et al., 2018; Wang et al., 2018a; Mirowski et al., 2017; 2018; Zamir et al., 2018). In this work, we concentrate on the recently proposed the Visionand-Language Navigation task (Anderson et al., 2018b)—asking an agent to carry out sophisticated natural-language instructions in a 3D environment. This task has application to fields such as robotics; in contrast to traditional map-based navigation systems, navigation with instructions provides a flexible way to generalize across different environments.
  A few approaches have been proposed for the VLN task. For example, Anderson et al. (2018b) address the task in the form of a sequence-to-sequence translation model. Yu et al. (2018) introduce a guided feature transformation for textual grounding. Wang et al. (2018b) present a planned-head module by combing model-free and model-based reinforcement learning approaches. Recently, Fried et al. (2018) propose to train a speaker to synthesize new instructions for data augmentation and further use it for pragmatic inference to rank the candidate routes. These approaches leverage attentional mechanisms to select related words from a given instruction when choosing an action, but those agents are deployed to explore the environment without knowing about what progress has been made and how far away the goal is. In this paper, we propose a self-monitoring agent that performs co-grounding on both visual and textual inputs and constantly monitors its own progress toward the goal as a way of regularizing the textual grounding.
  Visual and textual grounding. Visual grounding learns to localize the most relevant object or region in an image given linguistic descriptions, and has been demonstrated as an essential component for a variety of vision tasks like image captioning (Hu et al., 2016; Rohrbach et al., 2016; Lu et al., 2018), visual question answering (Lu et al., 2016b; Agrawal et al., 2018), relationship detection (Lu et al., 2016a; Ma et al., 2018) and referral expression (Nagaraja et al., 2016; Gavrilyuk et al., 2018). In contrast to identifying regions or objects, we perform visual grounding to locate relevant images (views) in a panoramic photo constructed by stitching multiple images with the aim of choosing which direction to go. Extensive efforts have been made to ground language instructions into a sequence of actions (MacMahon et al., 2006; Branavan et al., 2009; Vogel & Jurafsky, 2010; Tellex et al., 2011; Artzi & Zettlemoyer, 2013; Andreas & Klein, 2015; Mei et al., 2016; Cohn et al., 2016; Misra et al., 2017). These early approaches mainly emphasize the incorporation of structural alignment biases between the linguistic structure and sequence of actions (Mei et al., 2016; Andreas & Klein, 2015), and assume the agents are in relatively easy environment where limited visual perception is required to fulfill the instructions.
sections.heading: 5 CONCLUSION
sections.text: We introduce a self-monitoring agent which consists of two complementary modules: visual-textual co-grounding module and progress monitor. The visual-textual co-grounding module locates the instruction completed in the past, the instruction needed in the next action, and the moving direction from surrounding images. The progress monitor regularizes and ensures the grounded instruction correctly reflects the progress towards the goal by explicitly estimating the completeness of instruction-following. This estimation is conditioned on the positions and weights of grounded instruction. Our approach sets a new state-of-the-art performance on the standard Room-to-Room dataset on both seen and unseen environments. While we present one instantiation of self-monitoring for a decision-making agent, we believe that this concept can be applied to other domains as well.
sections.heading: ACKNOWLEDGMENTS
sections.text: This research was partially supported by DARPAs Lifelong Learning Machines (L2M) program, under Cooperative Agreement HR0011-18-2-001. We thank the authors from Fried et al. (2018), Ronghang Hu and Daniel Fried, for communicating with us and providing details of the implementation and synthetic instructions for fair comparison.
sections.heading: SUPPLEMENTARY MATERIALS
sections.heading: COMPARISON WITH PRIOR ART WITHOUT BEAM SEARCH
sections.text: We provide the comparison with state of the arts without using beam search. The results are shown in Table 3. We can see that our proposed method outperformed existing approaches with a large margin on both validation unseen and test sets. Our method with greedy decoding for action selection improved the SR by 9% and 8% on validation unseen and test set. When using progress inference for action selection, the performance on the test set significantly improved by 5% compared to using greedy decoding, yielding 13% improvement over the best existing approach.
sections.heading: IMPLEMENTATION DETAILS
sections.text: Image feature. Similar to previous work, we use the pre-trained ResNet-152 on ImageNet to extract image features. Each image feature is thus a 2048-d vector. The embedded feature vector for each navigable direction is obtained by concatenating an appearance feature with a 4-d orientation feature [sinφ; cosφ; sinθ; cosθ], where φ and θ are the heading and elevation angles. Following the work in Fried et al. (2018), the 4-dim orientation features are tiled 32 times, resulting a embedding feature vector with 2176 dimension.
  Network architecture. The embedding dimension for encoding the navigation instruction is 256. We use a dropout layer with ratio 0.5 after the embedding layer. We then encode the instruction using a regular LSTM, and the hidden state is 512 dimensional. The MLP g used for projecting the raw image feature is BN −→ FC −→ BN −→ Dropout −→ ReLU . The FC layer projects the 2176-d input vector to a 1024-d vector, and the dropout ratio is set to be 0.5. The hidden state of the LSTM used for carrying the textual and visual information through time in Eq. 1 is 512. We set the maximum length of instruction to be 80, thus the dimension of the attention weights of textual grounding αt is also 80. The dimension of the learnable matrices from Eq. 2 to 5 are: Wx ∈ R512×512,Wv ∈ R512×1024,Wa ∈ R1024×1024,Wh ∈ R1536×512, andWpm ∈ R592×1. Training. We use ADAM as the optimizer. The learning rate is 1e − 4 with batch size of 64 consistently through out all experiments. When using beam search, we set the beam size to be 15. We perform categorical sampling during training for action selection.
sections.heading: SUBMISSION TO VISION AND LANGUAGE NAVIGATION CHALLENGE
sections.text: For evaluating our proposed approach on the unseen test set, we participate in the Vision and Language Navigation challenge and submitted our result with the full proposed approach to the test server. We achieved 61% success rate and ranked #1 on the test server at the time of writing.
  We follow the submission guidelines, where picking the highest confidence trajectory from multiple trials for each instruction is not permissible. This means that using the beam search for competing and selecting a final trajectory is not allow directly. Similar to the submission from SpeakerFollower (Fried et al., 2018), we record all the viewpoints traversed during the beam search process. The final agent traverses through all recorded trajectories by first reaching the end of one trajectory and backtracking to the shared viewpoint with the next trajectory. This means that the agent could backtrack to the start point during this process. The trajectories are however logged according to the
  closest previous trajectory, so that when a single agent traverses through all recorded trajectories, the overhead for switching from one trajectory to another can be reduced significantly. The final selected trajectory from beam search is then lastly logged to the trajectory. This therefore yields exactly the same success rate and navigation error, as the metrics are computed according to the last viewpoint from a trajectory.
sections.heading: QUALITATIVE RESULTS
sections.text: We provide and discuss additional qualitative results on the self-monitoring agent navigating on seen and unseen environments. We first discuss four successful examples in Fig. 5 and 6, and followed by two failure examples in Fig. 7.
sections.heading: SUCCESSFUL EXAMPLES
sections.text: In Fig. 5 (a), at the beginning, the agent mostly focuses on ”walk up” for making the first movement. While the agent keeps its attention on ”walk up” as completed instruction or ongoing action, it shifts the attention on instruction to ”turn right” as it walks up the stairs. Once it reached the top of the stairs, it decides to turn right according to the grounded instruction. Once turned right, we can again see that the agent pays attention on both the past action ”turn right” and next action ”walk straight to bedroom”. The agent continues to do so until it decides to stop by grounding on the word ”stop”.
  In Fig. 5 (b), the agent starts by focusing on both ”enter bedroom from balcony” and ”turn left” to navigate. It correctly shifts the attention on textual grounding on the following instruction. Interestingly, the given instruction ”walk straight across rug to room” at step 3 is ambiguous since there are two rooms across the rug. Our agent decided to sneak out of the first room on the left and noticed that it does not match with the description from instruction. It then moved to another room across the rug and decided to stop because there is a rug inside the room as described.
  In Fig. 6 (a), the given instruction is ambiguous as it only asks the agent to take actions around the stairs. Since there are multiple duplicated actions described in the instruction, e.g. ”walk up” and ”turn left”, only an agent that is able to precisely follow the instruction step-by-step can successfully complete the task. Otherwise, the agent is likely to stop early before it reaches the goal. The agent also needs to demonstrate its ability to assess the completeness of instruction-following task in order to correctly stop at the right amount of repeated actions as described in the instruction.
  In Fig. 6 (b), at the beginning (step 0), the agent only focuses on ’left’ for making the first movement (the agent is originally facing the painting). We can see that at each step, the agent correctly focuses on parts of the instruction for making every movements, and it finally believes that the instruction is completed (attention on the last sentence period) and stopped.
sections.heading: FAILURE EXAMPLES
sections.text: In Fig. 7 (a) step 1, although the attention on instruction correctly focused on ”take a left” and ”go down”, the agent failed to follow the instruction and was not able to complete the task. We can however see that the progress monitor correctly reflected that the agent did not follow the given instruction successfully. The agent ended up stopping with progress monitor reporting that only 16% of the instruction was completed.
  In Fig. 7 (b) step 2, the attention on instruction only focuses on ”go down” and thus failed to associate the ”go down steps” with the stairs previously mentioned in ”turn right to stairs”. The agent was however able to follow the rest of the instruction correctly by turning right and stopping near a mirror. Note that, different from Fig. 7 (a), the final estimated completeness of instruction-following from progress monitor is much higher (16%), which indicates that the agent failed to be aware that it was not correctly following the instruction.
title: IARY PROGRESS ESTIMATION