Training a referring expression comprehension (ReC) model for a new visual domain requires collecting referring expressions, and potentially corresponding bounding boxes, for images in the domain. While large-scale pre-trained models are useful for image classification across domains, it remains unclear if they can be applied in a zero-shot manner to more complex tasks like ReC. We present ReCLIP, a simple but strong zero-shot baseline that repurposes CLIP, a state-of-the-art large-scale model, for ReC. Motivated by the close connection between ReC and CLIP's contrastive pre-training objective, the first component of ReCLIP is a region-scoring method that isolates object proposals via cropping and blurring, and passes them to CLIP. However, through controlled experiments on a synthetic dataset, we find that CLIP is largely incapable of performing spatial reasoning off-the-shelf. Thus, the second component of ReCLIP is a spatial relation resolver that handles several types of spatial relations. We reduce the gap between zero-shot baselines from prior work and supervised models by as much as 29% on RefCOCOg, and on RefGTA (video game imagery), ReCLIP's relative improvement over supervised ReC models trained on real images is 8%. Visual referring expression comprehension (ReC) the task of localizing an object in an image given a textual referring expressionhas applications in a broad range of visual domains. For example, ReC is useful for guiding a robot in the real world (Shridhar et al., 2020) and also for creating natural language interfaces for software applications with visuals (Wichers et al., 2018). Though the task is the same across domains, the domain shift is problematic for supervised referring expression models, as shown in Figure 1: the same simple  This work was done while Sanjay, Will, and Matt were affiliated with AI2. Collecting task-specific data in each domain of interest is expensive. Weakly supervised ReC (Rohrbach et al., 2016) partially addresses this issue, since it does not require the ground-truth box for each referring expression, but it still assumes the availability of referring expressions paired with images and trains on these. Given a large-scale pretrained vision and language model and a method for doing ReC zero-shoti.e. without any addi-5198 Figure 2: Overview of ReCLIP. tional trainingpractitioners could save a great deal of time and effort. Moreover, as pre-trained models have become more accurate via scaling (Ka-plan et al., 2020), fine-tuning the best models has become prohibitively expensiveand sometimes infeasible because the model is offered only via API, e.g. GPT-3 (Brown et al., 2020). Pre-trained vision and language models like CLIP (Radford et al., 2021) achieve strong zero-shot performance in image classification across visual domains (Jia et al., 2021) and in object detection (Gu et al., 2021), but the same success has not yet been achieved in tasks requiring reasoning over vision and language. For example, Shen et al. (2021) show that a straightforward zero-shot approach for VQA using CLIP performs poorly. Specific to ReC, Yao et al. (2021) introduce a zero-shot approach via Colorful Prompt Tuning (CPT), which colors object proposals and references the color in the text prompt to score proposals, but this has low accuracy. In both of these cases, the proposed zero-shot method is not aligned closely enough with the model's pre-training task of matching naturally occurring images and captions. In this work, we propose ReCLIP, a simple but strong new baseline for zero-shot ReC. ReCLIP, illustrated in Figure 2, has two key components: a method for scoring object proposals using CLIP and a method for handling spatial relations between objects. Our method for scoring region proposals, Isolated Proposal Scoring (IPS), effectively reduces ReC to the contrastive pre-training task used by CLIP and other models. Specifically, we propose to isolate individual proposals via cropping and blurring the images and to score these isolated proposals with the given expression using CLIP. To handle relations between objects, we first consider whether CLIP encodes the spatial information necessary to resolve these relations. We show through a controlled experiment on CLEVR images (Johnson et al., 2017) that CLIP and another pre-trained model ALBEF (Li et al., 2021) are unable to perform its pre-training task on examples that require spatial reasoning. Thus, any method that solely relies on these models is unlikely to resolve spatial relations accurately. Consequently, we propose spatial heuristics for handling spatial relations in which an expression is decomposed into subqueries, CLIP is used to compute proposal probabilities for each subquery, and the outputs for all subqueries are combined with simple rules. On the standard RefCOCO/g/+ datasets (Mao et al., 2016; Yu et al., 2016), we find that ReCLIP outperforms CPT (Yao et al., 2021) by about 20%. Compared to a stronger GradCAM (Selvaraju et al., 2017) baseline, ReCLIP obtains better accuracy on average and has less variance across object types. Finally, in order to illustrate the practical value of zero-shot grounding, we also demonstrate that our zero-shot method surpasses the out-of-domain performance of state-of-the-art supervised ReC models. We evaluate on the RefGTA dataset (Tanaka et al., 2019), which contains images from a video game (out of domain for models trained only on real photos). Using ReCLIP and an object detector trained outside the target domain, we outperform UNITER-Large (Chen et al., 2020) (using the same proposals) and MDETR (Kamath et al., 2021) by an absolute 4.5% (relative improvement of 8%). zero-shot spatial reasoning performance, and (3) a comparison of our zero-shot ReC performance with the out-of-domain performance of state-of-the-art fully supervised ReC systems. 1 2 Background In this section, we first describe the task at hand (2.1) and introduce CLIP, the pre-trained model we primarily use (2.2). We then describe two existing methods for scoring region proposals using a pre-trained vision and language model: colorful prompt tuning (2.3) and GradCAM (2.4). In referring expression comprehension (ReC), the model is given an image and a textual referring expression describing an entity in the image. The goal of the task is to select the object (bounding box) that best matches the expression. As in much of the prior work on REC, we assume access to a set of object proposals b 1 , b 2 , ..., b n , each of which is a bounding box in the image. Task accuracy is measured as the percentage of instances for which the model selects a proposal whose intersection-over-union (IoU) with the ground-truth box is at least 0.5. In this paper, we focus on the zero-shot setting in which we apply a pre-trained model to ReC without using any training data for the task. The zero-shot approaches that we consider are general in that the only requirement for the pretrained model is that when given a query consisting of an image and text, it computes a score for the similarity between the image and text. In this paper, we primarily use CLIP (Radford et al., 2021). We focus on CLIP because it was pretrained on 400M image-caption pairs collected from the web 2 and therefore achieves impressive zero-shot image classification performance on a variety of visual domains. CLIP has an image-only encoder, which is either a ResNet-based architecture (He et al., 2016) or a visual transformer (Dosovitskiy et al., 2021), and a text-only transformer. We mainly use the RN50x16 and ViT-B/32 versions of CLIP. The image encoder takes the raw image and produces an image representation x  R d , and the text transformer takes the 1 Our code is available at https://www.github. sequence of text tokens and produces a text representation y  R d . In CLIP's contrastive pretraining task, given a batch of N images and matching captions, each image must be matched with the corresponding text. The model's probability of matching image i with caption j is given by exp(  x i T y j ) / (cid:80) Nk =1 exp(  x i T y k ) , where  is a hyperparameter. 3 We now describe two techniques from prior work for selecting a proposal using a pre-trained model. The first baseline from prior work that we consider is colorful prompt tuning (CPT), proposed by Yao et al. (2021) 4 : they shade proposals with different colors and use a masked language prompt in which the referring expression is followed by in [MASK] color. The color with the highest probability from a pre-trained masked language model (MLM) (VinVL; (Zhang et al., 2021)) is then chosen. In order to apply this method to models like CLIP, that provide image-text scores but do not offer an MLM, we create a version of the input image for each proposal, where the proposal is transparently shaded in red. 5 Our template for the input text is [referring expression] is in red color. Since we have adapted CPT for non-MLM models, we refer to this method as CPT-adapted in the experiments. The second baseline from prior work that we consider is based on gradient-based visualizations, which are a popular family of techniques for understanding, on a range of computer vision tasks, which part(s) of an input image are most important to a model's prediction. We focus on the most popular technique in this family, GradCAM (Sel-varaju et al., 2017). Our usage of GradCAM follows Li et al. (2021), in which GradCAM is used to perform weakly supervised referring expression comprehension using the ALBEF model. In our setting, for a given layer in a visual transformer, we take the layer's class-token (CLS) attention matrix M  R h,w . The spatial dimensions h and w are dependent on the model's architecture and are generally smaller than the input dimensions 3 x i and y i are normalized before the dot product. of the image. Then the GradCAM is computed as G = M  LM , where L is the model's output logit (the similarity score for the image-text pair) and  denotes elementwise multiplication. The procedure for applying GradCAM when the visual encoder is a convolutional network is similar; in place of the attention matrix, we use the activations of the final convolutional layer. Next, we perform a bicubic interpolation on G so that it has the same dimensions as the input image. Finally, we compute for each proposal b i = ( x 1 , y 1 , x 2 , y 2 ) the score 1 A  (cid:80) x 2 i = x 1 (cid:80) y 2 j = y 1 G [ i, j ] , where A is the area of the image and  is a hyperparameter, and we choose the proposal with the highest score. ReCLIP consists of two main components: (1) a region-scoring method that is different from CPT and GradCAM and (2) a rule-based relation resolver. In this section, we first describe our region scoring method (3.1). However, using controlled experiments on a synthetic dataset, we find that CLIP has poor zero-shot spatial reasoning performance (3.2). Therefore, we propose a system that uses heuristics to resolve spatial relations (3.3). Our proposed method, which we call isolated proposal scoring , is based on the observation that ReC is similar to the contrastive learning task with which models like CLIP are pre-trained, except that rather than selecting one out of several images to match with a given text, we must select one out of several image regions. Therefore, for each proposal, we create a new image in which that proposal is isolated. We consider two methods of isolation  cropping the image to contain only the proposal and blurring everything in the image except for the proposal region. For blurring, we apply a Gaussian filter with standard deviation  to the image RGB values. Appendix A.2 provides an example of isolation by blurring. The score for an isolated proposal is obtained by passing it and the expression through the pre-trained model. To use cropping and blurring in tandem, we obtain a score s crop and s blur for each proposal and use s crop + s blur as the final score. This can be viewed as an ensemble of visual prompts, analogous to Radford et al. (2021)'s ensembling of text prompts. A key limitation in Isolated Proposal Scoring is that relations between objects in different proposals are not taken into account. For example, in Figure 2, the information about the spatial relationships among the cats is lost when the proposals are isolated. In order to use CLIP to decide which object has a specified relation to another object, the model's output must encode the spatial relation in question. Therefore, we design an experiment to determine whether a pre-trained model, such as CLIP, can understand spatial relations within the context of its pre-training task. We generate synthetic images using the process described for the CLEVR dataset (Johnson et al., 2017). These scenes include three shapesspheres, cubes, and cylindersand eight colorsgray, blue, green, cyan, yellow, purple, brown, red. In the text-pair version of our tasks, using the object attribute and position information associated with each image, we randomly select one of the pairwise relationships between objectsleft, right, front, or behindand construct a sentence fragment based on it. For example: A blue sphere to the left of a red cylinder. We also write a distractor fragment that replaces the relation with its opposite. In this case, the distractor would be A blue sphere to the right of a red cylinder. The task, similar to the contrastive and image-text matching tasks used to pre-train these models, is to choose the correct sentence given the image. As a reference point, we also evaluate on a control (non-spatial) task in which the correct text is a list of the scene's objects and the distractor text is identical except that one object is swapped with a random object not in the scene. For example, if the correct text is A blue sphere and a red cylinder, then the distractor text could be A blue sphere and a blue cylinder. single sentence fragment constructed as described above for the spatial and control (non-spatial) tasks and two images such that only one matches the text. Appendix B shows examples of these tasks. CLIP's performance on these tasks is shown in Table 1. Similar results for the pre-trained model ALBEF (Li et al., 2021) are shown in Appendix D.1 While performance on the control task is quite good, accuracy on the spatial task is not so different from random chance (50%). This indicates that the model scores of image-text pairs largely do not take spatial relations into account. Since CLIP lacks sensitivity to spatial relations, we propose to decompose complex expressions into simpler primitives. The basic primitive is a predicate applying to an object, which we use CLIP to answer. The second primitive is a spatial relation between objects, for which we use heuristic rules. Predicates A predicate is a textual property that the referent must satisfy. For example, the cat and blue airplane are predicates. We write P ( i ) to say that object i satisfies the predicate P . We model P as a categorical distribution over objects, and estimate p ( i ) = Pr[ P ( i )] with the pre-trained model using isolated proposal scoring ( 3.1). Relations We have already discussed the importance of binary spatial relations like the cat to the left of the dog for the ReC task. We consider seven spatial relations left , right , above , below , bigger , smaller , and inside . We write R ( i, j ) to mean that the relation R holds between objects i and j , and we use heuristics to determine the probability r ( i, j ) = Pr[ R ( i, j )] . For example, for left , we set r ( i, j ) = 1 if the center point of box i is to the left of the center point of box j and r ( i, j ) = 0 otherwise. C.1 describes all relation semantics. Superlative Relations We also consider superlatives, which refer to an object that has some relation to all other objects satisfying the same predicate, e.g. leftmost dog. We handle superlatives as a special case of relations where the empty second argument is filled by copying the predicate specifying the first argument. Thus, leftmost dog effectively finds the dog that is most likely to the left of other dog(s). Our set of superlative relation types is the same as our set of relation types, excluding inside . it procedurally. We first use spaCy (Honnibal and Johnson, 2015) to build a dependency parse for the expression. As illustrated in Figure 3, we extract a semantic tree from the dependency parse, where each noun chunk becomes a node, and dependency paths between the heads of noun chunks become relations between entities based on the keywords they contain. See C.2 for extraction details. In cases where none of our relation/superlative keywords occur in the text, we simply revert to the plain isolated proposal scoring method using the full text. In the tree, each node N contains a predicate PN and has a set of children; an edge ( N, N  ) between N and its child N  corresponds to a relation R N,N  . For example, as shown in Figure 3, a cat to the left of a dog would be parsed as a node containing the predicate a cat connected by the relation left to its child corresponding to a dog. We define  N ( i ) as the probability that node N refers to object i , and compute it recursively. For each node N , we first set  N ( i ) = p N ( i ) and then iterate through each child N  and update  N ( i ) as follows 6 :   N ( i )   N ( i ) (cid:88) j Pr (cid:2) R N,N  ( i, j )  PN  ( j ) (cid:3)   N ( i ) (cid:88) j r N,N  ( i, j )  N  ( j ) . The last line makes the simplifying assumption that all predicates and relations are independent. 7 To compute our final output, we ensemble the distribution  root for the root node with the output of plain isolated proposal scoring (with the whole input expression) by multiplying the proposal probabilities elementwise. This method gives us a principled way to combine predicates ( PN ) with spatial relational constraints ( R N,N  ) for each node N . We compare ReCLIP to other zero-shot methods on RefCOCOg (Mao et al., 2016), RefCOCO and RefCOCO+ (Yu et al., 2016). These datasets use images from MS COCO (Lin et al., 2014). RefCOCO and RefCOCO+ were created in a two-player game, and RefCOCO+ is designed to avoid spatial relations. RefCOCOg includes spatial relations and has longer expressions on average. For comparing zero-shot methods with the out-of-domain performance of models trained on COCO, we use RefGTA (Tanaka et al., 2019), which contains images from the Grand Theft Auto video game. All referring expressions in RefGTA correspond to people, and the objects (i.e. people) tend to be much smaller on average than those in RefCOCO/g/+. We use an ensemble of the CLIP RN50x16 and ViT-B/32 models (results for individual models are shown in Appendix G). We ensemble model outputs by adding together the logits from the two models elementwise before taking the soft-max. GradCAM's hyperparameter  controls the 6 Superlatives of a node are processed after all its relations. effect of the proposal's area on its score. We select  = 0 . 5 for all models based on tuning on the RefCOCOg validation set. We emphasize that the optimal value of  for a dataset depends on the size distribution of ground-truth objects. ReCLIP also has a hyperparameter, namely the standard deviation  . We try a few values on the RefCOCOg validation set and choose  = 100 , as we show in Appendix E.4, isolated proposal scoring has little sensitivity to  . As discussed by (Perez et al., 2021), zero-shot experiments often use labeled data for model selection. Over the course of this work, we primarily experimented with the RefCOCOg validation set and to a lesser extent with the RefCOCO+ validation set. For isolated proposal scoring, the main variants explored are documented in our ablation study (4.6). Other techniques that we tried, including for relation-handling, and further implementation details are given in Appendix E. 4.3 Results on RefCOCO/g/+ Table 2 shows results on RefCOCO, RefCOCO+, and RefCOCOg. ReCLIP is better than the other zero-shot methods on RefCOCOg and RefCOCO and on par with GradCAM on RefCOCO+. However, GradCAM has a much higher variance in its accuracy between the TestA and TestB splits of RefCOCO+ and RefCOCO. We note that GradCAM's 5203 hyperparameter  , controlling the effect of proposal size, was tuned on the RefCOCOg validation set, and RefCOCOg was designed such that boxes of referents are at least 5% of the image area (Mao et al., 2016). In the bottom portion of Table 2, we show that when this 5% threshold, a prior on object size for this domain, is used to filter proposals for both GradCAM and ReCLIP , ReCLIP performs on par with/better than GradCAM on TestA. ReCLIP's spatial relation resolver helps on RefCOCOg and RefCOCO but not on RefCOCO+, which is designed to avoid spatial relations. Next, we evaluate on RefGTA to compare our method's performance to the out-of-domain accuracy of two state-of-the-art fully supervised ReC models: UNITER-Large (Chen et al., 2020) and MDETR (Kamath et al., 2021). Like ReCLIP, UNITER takes proposals as input. 8 We show results using ground-truth proposals and detections from UniDet (Zhou et al., 2021), which is trained on the COCO, Objects365 (Shao et al., 2019), OpenImages (Kuznetsova et al., 2020), and Mapillary (Neuhold et al., 2017) datasets. Following the suggestion of the UniDet authors, we use the confidence threshold of 0.5. MDETR does not take proposals as input. Table 3 shows our results. For methods that take proposals (all methods except MDETR), we consider two evaluation settings using UniDet DT-P , in which the detected proposals are filtered to have only proposals whose predicted class label is per-son, and DT , in which all detected proposals are considered. ReCLIP's accuracy is more than 15% higher than the accuracy of UNITER-Large and roughly 5% more than that of MDETR. ReCLIP also outperforms GradCAM by about 20%, and the gap is larger when all UniDet proposals are considered. ReCLIP w/o relations is 1-2% better than ReCLIP in the settings with ground-truth proposals and filtered UniDet proposals. One possible reason for this gap is that the objects of relations in the expressions could be non-people entities. When 8 UNITER requires features from the bottom-up top-down attention model (Anderson et al., 2017). We use https://github.com/airsplay/ py-bottom-up-attention to compute the features for RefGTA. We trained UNITER models on RefCOCO+ and RefCOCOg using features computed from this repository. On the RefCOCO+ validation set, the resulting model has an accuracy roughly 0.4% less than that of a model trained and evaluated using the original features (when using ground-truth proposals). considering all UniDet proposals, the relation resolver in ReCLIP does not hurt accuracy much but also does not improve accuracy significantlyan additional challenge in this setting is that the number of proposals is dramatically higher. Appendix F shows qualitative examples of predictions on RefGTA. In order to determine how isolated proposal scoring (IPS) compares to GradCAM and CPT on other pre-trained models, we present results using ALBEF (Li et al., 2021). ALBEF offers two methods for scoring image-text pairsthe output used for its image-text contrastive (ITC) loss and the output used for its image-text matching (ITM) loss. The architecture providing the ITC output is very similar to CLIPhas only a shallow interaction between the image and text modalities. The ITM output is given by an encoder that has deeper interactions between image and text and operates on top of the ITC encoders' output. Appendix D provides more details. The results, shown in Table 4, show that with the ITC output, IPS performs better than GradCAM, but with the ITM output, GradCAM performs better. This suggests that IPS works well across models like CLIP and ALBEF ITC (i.e. contrastively pre-trained with shallow modality interactions) but that GradCAM may be better for models with deeper interactions. (a) ReCLIP is correct, while GradCAM is incorrect IPS achieves the highest accuracy for contrastively pre-trained models like CLIP. Figure 4a gives intuition for thisaside from an object's attributes, many referring expressions describe the local context around an object, and IPS focuses on this local context (as well as object attributes). Table 5 shows that using both cropping and blurring obtains greater accuracy than either alone. Error Analysis and Limitations Although ReCLIP outperforms the baselines that we consider, there is a considerable gap between it and supervised methods. The principal challenge in improving the system is making relation-handling more flexible. There are several object relation types Isolation type RefCOCOg RefCOCO+ Crop 54.43 41.28 Blur 55.96 47.23 max( Crop,Blur ) 55.76 44.55 Crop+Blur 57.70 47.43 Table 5: Ablation study of isolation types used to score proposals on Val splits of RefCOCOg/RefCOCO+, using detections from MAttNet (Yu et al., 2018). that our spatial relation resolver cannot handle; for instance, those that involve counting: the second dog from the right. Another challenge is in determining which relations require looking at multiple proposals. For instance, ReCLIP selects a proposal corresponding to the incorrect noun chunk in Figure 4b because the relation resolver has no rule for splitting an expression on the relation with. Depending on the context, relations like with may or may not require looking at multiple proposals, so handling them is challenging for a rule-based system. In the RefCOCO+ validation set, when using detected proposals, there are 75 instances for which ReCLIP answers incorrectly but ReCLIP w/o relations answers correctly. We categorize these instances based on their likely sources of error: 4 instances are ambiguous (multiple valid propos-als), in 7 instances the parser misses the head noun chunk, in 14 instances our processing of the parse leads to omissions of text when doing isolated proposal scoring (e.g. in girl sitting in back, the only noun chunk is girl, so this is the only text used during isolated proposal scoring), 52 cases in which there is an error in the execution of the heuristic (e.g. our spatial definition of a relation does not match the relation in the instance). (There are 2 instances for which we mark 2 categories.) The final category (execution) includes several kinds of errors, some examples of which are shown in Appendix F. 5 Related Work Referring expression comprehension Datasets for ReC span several visual domains, including photos of everyday scenes (Mao et al., 2016; Kazemzadeh et al., 2014), video games (Tanaka et al., 2019), objects in robotic context (Shridhar et al., 2020; Wang et al., 2021), and webpages (Wichers et al., 2018). Spatial heuristics have been used in previous work (Moratz and Tenbrink, 2006). Our work is also related to Krishnamurthy and Kollar (2013), 5205 which similarly decomposes the reasoning process into a parsing step and visual execution steps, but the visual execution is driven by learned binary classifiers for each predicate type. In the supervised setting, prior work shows that using an external parser, as we do, leads to lower accuracy than training a language module jointly with the remainder of the model (Hu et al., 2017). There is a long line of work in weakly supervised ReC, where at training time, pairs of referring expressions and images are available but the ground-truth bounding boxes for each expression are not (Rohrbach et al., 2016; Liu et al., 2019; Zhang et al., 2018, 2020; Sun et al., 2021). Our setting differs from the weakly supervised setting in that the model is not trained at all on the ReC task. Sadhu et al. (2019) discuss a zero-shot setting different from ours in which novel objects are seen at test time, but the visual domain stays the same. Pre-trained vision and language models Early pre-trained vision and language models (Tan and Bansal, 2019; Lu et al., 2019; Chen et al., 2020) used a cross-modal transformer (Vaswani et al., 2017) and pre-training tasks like masked language modeling, image-text matching, and image feature regression. By contrast, CLIP and similar models (Radford et al., 2021; Jia et al., 2021) use a separate image and text transformer and a contrastive pre-training objective. Recent hybrid approaches augment CLIP's architecture with a multi-modal transformer (Li et al., 2021; Zellers et al., 2021). Zero-shot application of pre-trained models Models pre-trained with the contrastive objective have exhibited strong zero-shot performance in image classification tasks (Radford et al., 2021; Jia et al., 2021). Gu et al. (2021) use CLIP can be to classify objects by computing scores for class labels with cropped proposals. Our IPS is different in that it isolates proposals by both cropping and blurring . Shen et al. (2021) show that a simple zero-shot application of CLIP to visual question answering performs almost on par with random chance. Yao et al. (2021) describe a zero-shot method for ReC based on a pre-trained masked language model (MLM); we show that their zero-shot results and a version of their method adapted for models pre-trained to compute image-text scores (rather than MLM) are substantially worse than isolated proposal scoring and GradCAM. We present ReCLIP, a zero-shot method for referring expression comprehension (ReC) that decomposes an expression into subqueries, uses CLIP to score isolated proposals against these subqueries, and combines the outputs with spatial heuristics. ReCLIP outperforms zero-shot ReC approaches from prior work and also performs well across visual domains: ReCLIP outperforms state-of-the-art supervised ReC models, trained on natural images, when evaluated on RefGTA. We also find that CLIP has low zero-shot spatial reasoning performance, suggesting the need for pre-training methods that account more for spatial reasoning. Recent work has shown that pre-trained vision and language models suffer from biases such as gender bias (Ross et al., 2021; Srinivasan and Bisk, 2021). Agarwal et al. (2021) provide evidence that CLIP has racial and other biases, which makes sense since CLIP was trained on data collected from the web and not necessarily curated carefully. Therefore, we do not advise deploying our system directly in the real world immediately. Instead, practitioners interested in this system should first perform analysis to measure its biases based on previous work and attempt to mitigate them. We also note that our work relies heavily on a pre-trained model whose pre-training required a great deal of energy, which likely had negative environmental effects. That being said our zero-shot method does not require training a new model and in that sense could be more environmentally friendly than supervised ReC models (depending on the difference in the cost of inference). We thank the Berkeley NLP group and Med-hini Narasimhan for helpful comments. We thank Michael Schmitz for help with AI2 infrastructure. This work was supported in part by DoD, including DARPA's LwLL, and/or Se-maFor programs, and Berkeley Artificial Intelligence Research (BAIR) industrial alliance programs. Sameer Singh was supported in part by the National Science Foundation grant #IIS-1817183 and in part by the DARPA MCS program under Contract No. N660011924033 with the United States Office Of Naval Research.
abstain abstain method abstain result abstain method abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain method objective objective result abstain objective result abstain objective method result abstain objective method abstain abstain method abstain method method method abstain abstain method other abstain abstain abstain method abstain abstain method abstain abstain method method method method abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain method method abstain abstain method abstain abstain abstain abstain abstain method abstain method abstain abstain method other abstain method method abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain result abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain other abstain other abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result method abstain result abstain abstain abstain abstain method abstain other other other other other