AI systems embodied in the physical world face a fundamental challenge of partial observability; operating with only a limited view and knowledge of the environment. This creates challenges when AI systems try to reason about language and its relationship with the environment: objects referred to through language (e.g. giving many instructions) are not immediately visible . Actions by the AI system may be required to bring these objects in view. A good benchmark to study this challenge is Dynamic Referring Expression Recognition ( dRER ) task where the goal is to find a target location by dynamically adjusting the field of view (FoV) in a partially observed 360  scenes. In this paper, we introduce HOLM, H allucinating O bjects with L anguage M odels, to address the challenge of partial observability. HOLM uses large pre-trained language models (LMs) to infer object hallucinations for the unobserved part of the environment. Our core intuition is that if a pair of objects coappear in an environment frequently, our usage of language should reflect this fact about the world. Based on this intuition, we prompt language models to extract knowledge about object affinities which gives us a proxy for spatial relationships of objects. Our experiments show that HOLM performs better than the state-of-the-art approaches on two datasets for dRER ; allowing to study generalization for both indoor and outdoor settings. One of the fundamental challenges in building AI systems physically present in the world is addressing the issue of partial observability, the phenomenon where the entire state of the environment is not known or available to the system. People cope with partial observability by reasoning about what is not immediately visible (see example in Figure 1). People combine their general knowledge about the world and adapt their knowledge     Figure 1: Illustration of our main contribution: Hallucinating Objects . to specific contexts (Torralba et al., 2006). General knowledge about kitchens can help to know approximately where to look for pans or utensils in a kitchen that has never been seen before. How can an AI system build general knowledge about objects and their environment to help with a similar task? Even more interestingly, can we gather this information from language, using readily available resources such as language models trained on a large collection of unlabeled text? In this paper, we introduce a method called HOLM, H allucinating O bjects with L anguage M odels, for reasoning about the unobserved parts of the environment. Inspired by the recent successes of large pre-trained language models (LM) extracting knowledge about the real world, we propose a methodology based on spatial prompts to extract knowledge from language models about object. HOLM extracts spatial knowledge about objects in the form of affinity scores, i.e., how often a pair of objects are observed together. This knowledge of objects are combined with observed spatial 5440 Figure 2: Illustration of the dRER task with an example of language instruction and its recognition in four steps. The agent adjusts its FoV by looking at different directions and navigate on the graph in the spherical view. Note that objects mentioned in bold in the instruction are not visible at all until timestep 4. Thus, the agent needs to reason about possible locations of the mentioned object using its partial view of the scene. layout to hallucinate what might appear in the unobserved part of the scene. We evaluate our HOLM approach on Dynamic Referring Expression Recognition ( dRER ) task where the goal is to find a target location by dynamically adjusting the field of view (FoV) in partially observed 360  scenes. We examine how HOLM compares with the state-of-the-art approaches on two publicly available datasets to study generalization for both indoor and outdoor settings. dRER task is designed to localize a target location in a dynamically observed 360  scene given natural language instruction. Unlike conventional referring expression recognition, which refers to an object in a static visual input, in dRER , only a small part of the scene is visible in a field of view. However, the system can adjust the field of view to find the described point in the scene. In Figure 2, we illustrate the dRER task and motivate our method. On top, natural language instruction is given. In the middle, the spherical view of the scene is illustrated  the agent explores only some portion of a 360  scene. FoVs on the sphere represented as square nodes form a graph. By navigating to a neighboring node , the agent adjusts its FoV and observes a different view of the scene. Note that objects mentioned in the instruction oven and range hood are not visible until the fourth timestep. However, we can reason about where to look using visible objects such as the air vent or the fridge. Thus, to perform well on this task, it is essential to reason about where objects might appear. (cid:104)S , A , P s , r (cid:105) where S is the visual state space, A is the discrete action space 1 , P s is the unknown environment probability distribution from which the next state is drawn, and r  R is the reward function. For a time step t , the agent observes an image s t  S , and performs and action a t  A . As a result of this action, the environment generates a new observation s t +1  P s (  | s t , a t ) as the next state. This interaction continues sequentially and ends when the agent performs a special STOP action or a pre-defined maximum episode length is reached. The resolution process is successful if the agent ends the episode at the target location. In dRER , instructions are represented as N sequence of sentences represented as x = { x i } Ni =1 . Each instruction sentence x i consists of a sequence of L i words, x i = [ x i, 1 , x i, 2 , ..., x i,L i , ] . The training dataset DE = {X , T } consists of M pairs of the instruction sequence x  X and its corresponding expert trajectory   T . The agent learns to navigate by learning a policy  via maximum like-1 For computational efficiency, we picked discrete action space. max  L  ( X , T ) , where L  ( X , T ) = log   ( T |X ) L  ( X , T ) = 1 MM (cid:88) k =1 log   (  k | x k ) (1) In dRER , the system observes the current FoV and does not see the resulting FoV before taking any actions. Thus, it is essential to reason what might appear in a future observation using what is currently visible to the system. Our core intuition is that objects visible in the current FoV and their locations in the FoV give us a clue about what might appear if a particular action is taken. Here, we propose an approach for reasoning about future observations using what is visible and some background knowledge of objects. Let us go through the illustration in Figure 3 to explain our HOLM method. In the top panel, we feed spatial prompts to pre-trained language models to extract knowledge about objects in the form of affinity scores. In the bottom panel, we see the input of the system where there are natural language instructions, an FoV of the scene, and detected objects. Next, we calculate which objects are relevant to each action. For instance, couch detections are on the right side; 5442 thus, they are relevant to the right action. Similarly, the fridge is relevant for the left action because it is on the left side. Then on the third step, using the affinity score of a pair of objects, we predict what might appear after performing an action. For right action, our model hallucinates a tv and tv-stand might appear because the couch and tv have a high affinity score according to the LM. Language models process a large amount of text to learn regularities in natural language. They do so by predicting the next word or masked token given a sequence of words. Our intuition is that objects that frequently appear in an environment close to each other will have similar language usage. Thus, we hypothesize that language models' capability of learning affinity scores of words in language also reflects objects' spatial properties. In Figure 3's top panel, we illustrate how we extract this capability. We query language models trained on a large amount of free-form text with spatial relationship prompts. These spatial prompts aim to capture the usage of words when they appear together in the world. An example of these prompt templates is Near the o 1 , there is ___ where o 1  O is an object label where O is a set of object labels. If object o 1 co-occurs with o 2 with high frequency, the language model would provide a high probability for the phrase Near the o 1 , there is o 2 . Using all pairs in O and K 2 spatial templates, we generate queries q . We then calculate affinity scores C o 1 ,o 2 , i.e., observing o 2 when o 1 is present as follows: C o 1 ,o 2 = K (cid:88) i =1 p LM ( o 2 | q i ) (2) Where p LM ( o 2 | q ) is a language model that calculates the probability of observing a token o 2 given a prefix sequence of tokens q . Our main idea behind HOLM is to reason about what might be observed in a future observation by combining (1) which objects are visible in the current observation and (2) what we know about the spatial properties of those objects. We explain the details of our approach in this section. Let p a  R | O | be the vector of probabilities of observing an object among a set of all objects O 2 Please see Appendix A.1 for the full list of spatial prompt templates. Where p FoV  R | O | is a vector of confidence values for objects detected in the current FoV. We use an off-the-shelf object detection system (Anderson et al., 2018a) to calculate p FoV . C is the affinity scores of size | O |  | O | . C represents how often a pair of object appear in a spatial relationship and represents the background knowledge of objects. 1 a  { 0 , 1 } | O | is a binary vector representing spatially related objects for a direction a . This vector is calculated with an indicator function to determine whether an object is spatially related to action a . We calculate the indicator function as follows. First, we separate the FoV into 4 imaginary regions called quadrants where each quadrant determines how a region in observed FoV is spatially relevant for canonical directions (i.e., up, down, left, right). In other words, quadrants are hot-spots for each direction i.e., the left side of the image is more relevant to the right side of the image if we are interested in what might appear on the left. For 8 directions (left, right, down, up, down-left, downright, up-left, up-right), we calculate how much each objects' bounding box overlaps with these quadrants. If intersection-over-union is above a fixed threshold we keep this object for the hallucination process. We designed our experiments to study and evaluate our proposed HOLM approach under five different research questions. RQ1: What is the performance of HOLM when compared to other state-of-the-art approaches? RQ2: what is the impact of LM as a source of knowledge for HOLM when compared to other more conventional sources (e.g., images)? RQ3: How essential are external sources of data for learning knowledge about objects compared to in domain data? RQ4: How accurate is HOLM for predicting objects in future observations? RQ5: How do annotation-free language-based knowledge sources i.e., LMs and word embeddings compare for HOLM? The following section explains the details of experimental setup. Our results are presented and discussed in Section 4.2. To study the research questions previously mentioned, we used two publicly available datasets and state-of-the-art methods as baselines to compare with. Datasets. We selected the following two datasets to see if our method generalizes to both indoor and outdoor settings. The Refer360  dataset (Cirik et al., 2020) consists of 17K natural language instructions and ground-truth trajectory pairs for localizing a target point in 360  scenes. The ground-truth trajectories are annotated by human annotators in the form of successive FoVs in partially observed 360  scenes. The dataset uses a subset of the SUN360 dataset (Xiao et al., 2012) as the source of scenes and these scenes are from both indoor and two outdoor locations. Touchdown (Chen et al., 2018) consists of 9K natural language instruction and ground-truth location pairs for 360  scenes on Google Streetview. Unlike the Refer360  dataset, Touchdown does not have expert trajectories  only expert predictions for the target location are provided. Thus, we generated ground-truth trajectories by calculating shortest path trajectories between a randomly selected starting point 3 and the target location. The Self Monitoring Navigation Agent (SMNA) (Ma et al., 2019) model is trained with a co-grounding module where both visual and textual input is attended at the same time. The agent also measures its progress with a progress monitor module. FAST (Ke et al., 2019) stands for Frontier Aware Search with backTracking. The FAST model learns to score partial trajectories of an agent for efficiently backtracking to a previous location after a mistake. Speaker-Follower (Fried et al., 2018) uses a sequence-to-sequence speaker model to rerank a follower model's candidate trajectories. This pragmatic reasoning model has been shown to improve navigation agents' performance significantly. 3 Following (Cirik et al., 2020), we set the initial random point to be a fix heading and random yaw. LingUNet (Misra et al., 2018) is an image-to-image encoder-decoder model for learning image-to-image mappings conditioned on language. We should emphasize that, unlike the previous methods, LingUNet is not a navigation model; instead, it predicts regions over an image. RANDOM agent randomly picks an action. STOP agent predicts the starting FoV as the target FoV. For a fair comparison, the same model was used as the basis for all the compared models. For our proposed approach HOLM is used to enhance the SMNA baseline by hallucinating objects for unseen regions. After getting object hallucinations for each neighboring FoVs, we use the sum of word embeddings for object labels as the input representation for the neighboring FoV. In the oracle Next FoV scenario, we use ground-truth FoVs to do the same process. For a fair comparison, we use SMNA as the base agent for learning to recover from a mistake during navigation process with FAST and as the follower model for pragmatic reasoning with Speaker-Follower. Evaluation Metrics. Our main evaluation metric for methods is FoV accuracy: the percentage of the time the target location is visible in the final FoV. The FoV accuracy sets an upper bound on the localization accuracy for predicting the pixel location of the target point, i.e., if the target is not visible, it is impossible to predict the exact location. Thus, we focus on this metric to compare systems. Implementation. All models are trained for 100K iterations. We use Adam (Kingma and Ba, 2015) for optimization with a learning rate 0.0001 and weight decay parameter 0.0005 (Krogh and Hertz, 1992). For each model, we perform a grid-search over their hyperparameters (e.g., number of hidden units, number of layers, dropout rate) and pick the best performing model based on validation score 4 . All models are implemented using PyTorch (Paszke et al., 2019) and publicly available 5 . To speed up the training procedure, we used fixed a grid of FoVs for all 360  images where each FoV is connected to its neighboring FoVs. This grid forms the navigation graph depicted in 4 For Refer360  we use validation unseen split. Touchdown does not have seen-unseen distinction. 5 https://github.com/volkancirik/HOLM 5444 Method Oracle Refer360  Touchdown Stop Agent 14.1 0.0 Random Agent 12.1 6.8 SMNA (Ma et al., 2019) 27.1 45.9 + HOLM (this work) 32.2 49.8 SMNA (Ma et al., 2019) Next FoV 33.5 50.2 LingUNet* (Chen et al., 2018) Full Panorama 21.4 47.2 Table 1: FoV accuracy results for Refer360  and Touchdown with no hallucination baseline, best performing models, and Next FoV oracle model, i.e. the ability to look ahead for neighbor FoVs, and observing full 360  scenes. the Figure 2. We use 30  of separation between successive FoVs which provides enough overlap to reveal relevant information about successive FoVs yet distant enough so that the model needs to reason about future steps. We then pre-calculated the rectilinear projection of each of the FoVs on the grid for all scenes. In this section we present and discuss experimental results and analyses. (RQ1) HOLM Improves performance. Our main results are presented in Table 1. In the first row block, we see that simple non-learning baselines fail to perform on the dRER . In the second row block, we compare our method with the baseline where the agent does not have any visual input from the next FoVs. HOLM improves the baseline by hallucinating objects for the next FoVs. In the third row block, we provide results for oracle scenarios. For SMNA, we feed ground-truth FoV as the input of the system. This result sets the upper bound on HOLM, because it cannot achieve better hallucination than the ground-truth FoVs. However, HOLM achieves pretty close to this upper bound and show that it can provide useful predictions for this task. For LingUNet, we feed the full 360  scenes as the visual input. Since LingUNet is not a navigation agent i.e. predicts the target location using full 360  scenes, we calculate FoV accuracy by drawing an FoV around the prediction, which explains *'. In Table 2, we compare HOLM with FAST and Speaker-Follower methods, both of which use beam search. During the beam search, these methods use multiple trajectories while deciding on a trajectory. However, this is not plausible in a real-world scenario, i.e. a robot would not gen-Method Beam Search Refer360  Touchdown Baseline SMNA (Ma et al., 2019) 27.1 45.9 + HOLM (this work) +5.1 +3.9 + FAST (Ke et al., 2019) (cid:33) -6.4 +4.7 + Speaker-Follower (Fried et al., 2018) (cid:33) -4.6 -11.1 Table 2: FoV accuracy results for Refer360  and Touchdown for methods using beam search or single candidate trajectory. erate many trajectories before performing action. HOLM, on the other hand completes the task on a single trajectory while predicting possible future states. FAST improves SMNA for Touchdown but not for Refer360  , which might be due to the richness of scenes in Refer360  whereas in Touchdown , the scenes are always in the same domain. Speaker-Model's decreases the score for SMNA possibly due to the Speaker models' poor performance where the BLEU score is around 6. HOLM consistently improves for both datasets and does not perform any expensive look-ahead operations such as beam search. affinity scores compared to other sources. In Table 3, we compare several baseline methods for calculating the affinity scores. First, we use uniform (i.e., each object pair has the same affinity score) and identity (i.e., object x can only have affinity score with itself) baselines. We also study calculating affinity scores using data annotated by humans. First, we use object annotations in VisualGenome (Krishna et al., 2017). VisualGenome provides a large collection of fine-grained annotations for objects and their spatial relationships. Second, ideally we would like to use human annotations for calculating the affinity score. However, this requires annotation of | O | 2 annotations. Instead, as a proxy, we use WordNet (Miller, 1995), a knowledge-base hierarchy annotated by experts. We use NLTK (Bird et al., 2009) to calculate the WordNet similarity to extract the affinity scores between ob-5445 jects. XLM-based HOLM achieves the best results among these baselines. This result shows that without using human annotations, we can extract useful knowledge about objects using pre-trained LMs. information compared to task data. In Table 4, we compare methods that only use task data for object hallucination and HOLM with external sources such as pre-trained LM. For the second row in the table), we use the BUTD model (Anderson et al., 2018a) to annotate training images with object bounding boxes. Using bounding boxes of objects, we calculate affinity scores. For the third row in the table, we design a model that takes FoV and an object type as an input and predicts a direction (i.e., hallucinate where it might appear) as output. We pass the final feature map layer of 152-layer ResNet (He et al., 2016) as input to a 3-layer feed-forward neural network to predict objects that might appear in neighboring FoVs. This model achieves an F1 score of 40.3 for direction prediction. Both of these methods improve over the SMNA baseline but are worse than the pretrained LM. This result indicates that task data may have limitations, and external sources such as a pretrained LM may provide a signal for knowledge about objects. (RQ4) Accuracy of HOLM translates to dRER So far, we measure the performance of HOLM for the downstream dRER task. We can also measure how accurate HOLM is at predicting the presence of an object in neighboring FoVs. We annotate each neighboring ground-truth FoVs with detections from BUTD. If the p ia for object o i  O is above 1 | O | , we count that as a prediction of an object in the neighboring FoV after performing action a . In Table 5, we provide precision, recall, and F1 score for the performance of different methods for calculating affinity scores for HOLM. XLM achieves the best performance among the methods we compare. We conclude that the performance for the intrinsic task (i.e., predicting the presence of objects) translates to dRER performance. good sources of general knowledge of objects In Table 6, we compare word embedding methods and different language models. We use cosine similarities between pairs of objects to calculate the affinity scores. For language models, we compare Open AI's GPT3 (Brown et al., 2020) using their online API 6 . We use Transformers Library (Wolf et al., 2020) for RoBERTa (Liu et al., 2019c) and XLM (Conneau and Lample, 2019). All methods consistently improve over the baseline SMNA model, however, we achieve the best performance using XLM. This result indicates that we can extract useful knowledge about objects with methods relying on large amount of unlabeled text. Our work on dRER is closely related to previous studies focusing on Referring Expression Recognition (RER), Vision-and-Language Navigation (VLN), and methods we propose are related to pretraining language models for vision-and-language tasks, model-based reinforcement learning, and co-occcurrence modeling for computer vision. We review these studies in this section. RER is the task of localizing a target object or a point in an image described by a natural language expression. The most of existing datasets 6 https://beta.openai.com/ 5446 poses the task in 2D images with objects as being the target (Kazemzadeh et al., 2014; Yu et al., 2016; Mao et al., 2016; Strub et al., 2017; Liu et al., 2019a; Akula et al., 2020; Chen et al., 2020). Several lines of work are proposed to address RER (Mao et al., 2016; Nagaraja et al., 2016; Yu et al., 2016; Hu et al., 2016; Fukui et al., 2016; Luo and Shakhnarovich, 2017; Liu et al., 2017; Yu et al., 2017; Zhang et al., 2018; Zhuang et al., 2018; Deng et al., 2018; Yu et al., 2018; Cirik et al., 2018; Liu et al., 2019b). In Touchdown (Chen et al., 2018) and Refer360  (Cirik et al., 2020) the target is a point not an object in a 360  image. In the dRER setup, we also use 360  images of Touchdown and Refer360  , but we do not provide the full panoramic view of the scene. Instead, in a more realistic scenario, the agent observes a partial and dynamic view of the scene, i.e. the agent needs to adjust its FoV to find the target location. Closer to our work, in REVERIE (Qi et al., 2020b) an embodied setup is proposed where the agent needs to first navigate to a location where the target object is visible. Similar to Touchdown and Refer360  , at the final position, the full 360  view is visible to the agent. Unlike ours and similar to 2D image-based RER, the target is an object rather than a point in the scene. VLN is a vision-and-language task where an agent in a simulated environment observes a visual input and is given a natural language instruction to navigate to a target location. The earlier work (MacMahon et al., 2006; Shimizu and Haas, 2009; Chen and Mooney, 2011) studies the task with synthetic images or in a very small scale (Vogel and Jurafsky, 2010). Anderson et al. (2018b) proposes Room-to-room (R2R) benchmark and revisit VLN task with a modern look. In R2R, the agent observes panoramic scans of a house (Chang et al., 2017) and needs to carry out the natural language instruction. EnvDrop (Tan et al., 2019) model shows generalization to unseen environments by dropping visual features. PREVALENT (Hao et al., 2020) tackles the data sparsity problem with a pretraining scheme. Hong et al. (2021) show that a pre-trained multi-modal can be enhanced with a memory state for the VLN task by recurrently feeding a contextualized state feature after each time step. dRER also poses a navigation task where locations in physical space in VLN correspond to FoVs in a fixed location. In dRER , a trajectory of the agent corresponds to its resolution process for finding the goal location. Pre-trained models for Vision-and-Language has been recently studied after the huge success of transformer-based models (Vaswani et al., 2017) in NLP (Devlin et al., 2018; Liu et al., 2019c; Conneau and Lample, 2019; Sun et al., 2019b; Poerner et al., 2020; Raffel et al., 2020; Brown et al., 2020). Numerous studies extend these approaches to the multimodal domain (Tan and Bansal, 2019; Lu et al., 2019; Sun et al., 2019a; Su et al., 2020; Li et al., 2020; Qi et al., 2020a; Hu and Singh, 2021). They achieve the-state-of-the-art results in several tasks such as image captioning, text-to-image retrieval, or referring expression recognition. Our work differs from these studies in the sense that the previous approaches use large scaled paired image-text data (Chen et al., 2013; Divvala et al., 2014; Sadeghi et al., 2015; Radford et al., 2021; Jia et al., 2021) to learn efficient representations (Frome et al., 2013; Kottur et al., 2016) for visual and textual modalities whereas we are interested in spatial information learned in unimodal text representations. Language priors for vision were explored in recent studies. Lu et al. (2016) use word embeddings in a language module to learn a representation for a object-predicate-object triplet for visual relationship detection task. Kiela et al. (2019) propose an approach to extend pre-trained transformer-based LMs for multimodal tasks. Similarly, Lu et al. (2021); Tsimpoukelli et al. (2021) show that pre-trained LMs can be finetuned to perform well in few-shot settings for image classification and open-domain Visual Question Answering (Marino et al., 2019). Marino et al. (2021) also show that multimodal transformer architectures capture implicit knowledge for a pair of objects. Our work differs from these studies (1) we use only unimodal models, (2) we do not finetune models  we do not update models during training. The most similar work to ours, Scialom et al. (2020) show that pre-trained LMs can perform reasonably well on Visual Question Generating (Yang et al., 2015; Mostafazadeh et al., 2016) out of the box. One difference is that we use object labels rather than object features or the appearance of objects to query the language model; however, they use object features as a visual token to the language model. Prompts we use in our work shares similarities with prompts designed in PIQA (Paranjape et al., 2021), but our work is evaluated in a multimodal setup. In con-5447 trast, PIQA is evaluated for textual commonsense reasoning tasks. Hallucination idea is also related the work on predicting future observations in long horizons (Vil-legas et al., 2019) which has been studied in the context of learning planning (Hafner et al., 2019) and acquiring skills for control problems (Hafner et al., 2020), and efficient policy learning (Ha and Schmidhuber, 2018), and vision-and-language navigation (Koh et al., 2021). All these approaches are interested in longer horizons; however, in our work, we study predicting single-step future observation. More recent work (Hu et al., 2021; Rombach et al., 2021; Rockwell et al., 2021) study view synthesis from a single visual observation. Unlike these approaches, HOLM does not generate pixel-level views rather abstractions of views with object labels. Affinity scores are mainly studied in computer vision tasks in the form of object co-occurrences. Previous studies have shown that object co-occurrences are efficient representations of visual prior for object categorization for object segmentation (Rabinovich et al., 2007; Galleguil-los et al., 2008; Ladicky et al., 2010) and zero shot object-recognition (Mensink et al., 2014), and scene understanding (Wu et al., 2014). Our work differs from these studies: we do not calculate co-occurrence statistics, i.e. we do not count the frequency of times they appear together; instead, we calculate a probability measure using language models. In this paper, we introduced HOLM  a model that can extract prior knowledge about objects from LMs and hallucinate objects in future observations. Our experiments showed that HOLM approach improves over various baselines from the literature. Surprisingly, our model which used background knowledge from LMs outperformed models with knowledge from human-annotated data showing that LMs learn useful knowledge about the world without requiring any visual observations. We also showed that out approach generalizes to both indoor and outdoor scenarios. Our work has limitations in the following ways. First, the hallucination process solely conditions on the current field of view. However, the instruction and the previous observations are available to the system. Conditioning on these sources of information could improve the hallucination accuracy by getting more targeted information from the language model. Second, we assume a fixed lexicon of object labels for hallucination. For both the visual side i.e., the object detector, and the language side i.e., the language model, when an unknown object appears the system cannot use this object for hallucination. Another issue is the scalability, i.e, the affinity scores scale with O ( N 2 ) where N is the number of objects, which might be challenging when N is large. We hope the follow-up work could address these limitations. Future work will explore the use of background knowledge in other domains such as vision-and-language navigation (Anderson et al., 2018c) and dialog (Thomason et al., 2020). We also believe background knowledge of objects would be handy in complex scenarios such as manipulating objects in a simulated environment (Shridhar et al., 2020). Our method examines extracting background knowledge in a zero-shot manner. However, the literature shows that learning how to prompt could be helpful in finding better (Liu et al., 2021). We strictly compared unimodal approaches for hallucination. Future work extend our work by comparing multimodal models (Tan and Bansal, 2019; Lu et al., 2019; Sun et al., 2019a; Su et al., 2020; Li et al., 2020; Qi et al., 2020a; Hu and Singh, 2021). Another interesting direction would be to study the capability of transferring knowledge from indoor to outdoor settings and vise versa. Finally, the success of PREVALENT (Hao et al., 2020) and other pre-training approaches for VLN could stem from their ability to implicitly encode prior knowledge about objects. Hopefully, future studies examines this phenomenon. This material is based upon work partially supported by National Science Foundation awards 1722822 and 1750439, and National Institutes of Health awards R01MH125740, R01MH096951 and U01MH116925. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors, and no official endorsement should be inferred. We also thank anonymous reviewers of ACL Rolling Review for their valuable feedback.
abstain abstain abstain abstain abstain abstain abstain method result abstain abstain objective abstain abstain abstain method abstain objective abstain abstain abstain abstain abstain abstain objective method abstain abstain abstain objective abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain method other abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method method method method abstain method abstain method abstain abstain method method abstain method method abstain other abstain method method result abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain method method abstain abstain abstain method method method method method abstain abstain abstain abstain method method abstain method method abstain method method method method method abstain abstain objective abstain other other other other method other abstain other other other other other other other other other other other other other other other other other other other other objective other method method other other method other other other other objective method result result result method abstain abstain abstain method abstain abstain method abstain method method abstain method abstain abstain abstain abstain other other other