Visual referring expression recognition is a challenging task that requires natural language understanding in the context of an image. We critically examine RefCOCOg , a standard benchmark for this task, using a human study and show that 83.7% of test instances do not require reasoning on linguistic structure, i.e., words are enough to identify the target object, the word order doesn't matter. To measure the true progress of existing models, we split the test set into two sets, one which requires reasoning on linguistic structure and the other which doesn't. Additionally, we create an out-of-distribution dataset Ref-Adv by asking crowdworkers to perturb in-domain examples such that the target object changes. Using these datasets, we empirically show that existing methods fail to exploit linguistic structure and are 12% to 23% lower in performance than the established progress for this task. We also propose two methods, one based on contrastive learning and the other based on multi-task learning, to increase the robustness of ViLBERT, the current state-of-the-art model for this task. Our datasets are publicly available at https://github.com/ aws/aws-refcocog-adv . Visual referring expression recognition is the task of identifying the object in an image referred by a natural language expression (Kazemzadeh et al., 2014; Nagaraja et al., 2016; Mao et al., 2016; Hu et al., 2016). Figure 1 shows an example. This task has drawn much attention due to its ability to test a model's understanding of natural language in the context of visual grounding and its application in downstream tasks such as image retrieval (Young et al., 2014) and question answering (Antol et al., 2015; Zhu et al., 2016). To track  Work done in part while AA was intern at Amazon AI. progress on this task, various datasets have been proposed, in which real world images are annotated by crowdsourced workers (Kazemzadeh et al., 2014; Mao et al., 2016). Recently, neural models have achieved tremendous progress on these datasets (Yu et al., 2018; Lu et al., 2019). However, multiple studies have suggested that these models could be exploiting strong biases in these datasets (Cirik et al., 2018b; Liu et al., 2019). For example, models could be just selecting a salient object in an image or a referring expression without recourse to linguistic structure (see Figure 1). This defeats the true purpose of the task casting doubts on the actual progress. In this work, we examine RefCOCOg dataset (Mao et al., 2016), a popular testbed for evaluating referring expression models, using crowdsourced workers. We show that a large percentage of samples in the RefCOCOg test set indeed do not rely on linguistic structure (word order) of the expressions. Accordingly, we split RefCOCOg test set into two splits, Ref-Easy and Ref-Hard , where linguistic structure is key for recognition in the latter but not the former (  2). In addition, we create a new out-of-distribution 1 dataset called Ref-Adv using Ref-Hard by rewriting a referring expression 1 This is a contrast set according to Gardner et al. (2020) such that the target object is different from the original annotation (  3). We evaluate existing models on these splits and show that the true progress is at least 12-23% behind the established progress, indicating there is ample room for improvement (  4). We propose two new models, one which make use of contrastive learning using negative examples, and the other based on multi-task learning, and show that these are slightly more robust than the current state-of-the-art models (  5). RefCOCOg is the largest visual referring expression benchmark available for real world images (Mao et al., 2016). Unlike other referring expression datasets such as RefCOCO and RefCOCO+ (Kazemzadeh et al., 2014), a special care has been taken such that expressions are longer and diverse. We therefore choose to examine the importance of linguistic structure in RefCOCOg . Cirik et al. (2018b) observed that when the words in a referring expression are shuffled in random order, the performance of existing models on RefCOCOg drops only a little. This suggests that models are relying heavily on the biases in the data than on linguistic structure, i.e., the actual sequence of words. Ideally, we want to test models on samples where there is correlation between linguistic structure and spatial relations of objects, and any obscurity in the structure should lead to ambiguity. To filter out such set, we use humans. We randomly shuffle words in a referring expression to distort its linguistic structure, and ask humans to identify the target object of interest via predefined bounding boxes. Each image in RefCOCOg test set is annotated by five Amazon Mechanical Turk (AMT) workers and when at least three annotators select a bounding box that has high overlap with the ground truth, we treat it as a correct prediction. Following Mao et al. (2016), we set 0.5 IoU (intersection over union) as the threshold for high overlap. Given that there are at least two objects in each image, the optimal performance of a random choice is less than 50%. 2 However, we observe that human accuracy on distorted examples is 83.7%, indicating that a large portion of RefCOCOg test set is insensitive to linguistic structure. Based on this observation, we divide the test set into two splits for fine-grained evaluation of models: Ref-Easy contains samples insensitive 2 On average, there are 8.2 bounding boxes per image. Due to unintended annotation artifacts in RefCOCOg , it is still possible that models could perform well on Ref-Hard without having to rely on linguistic structure, e.g., by selecting frequent objects seen during training time. Essentially, Ref-Hard is an in-distribution split. To avoid this, we create Ref-Adv , an adversarial test set with samples that may be fall out of training distribution. We take each sample in Ref-Hard and collect additional referring expressions such that the target object is different from the original object. We chose the target objects which humans are most confused with when the referring expression is shuffled (as described in the previous section). For each target object, we ask three AMT workers to write a referring expression while retaining most content words in the original referring expression. In contrast to the original expression, the modified expression mainly differs in terms of the structure while sharing several words. For example, in Figure 1, the adversarial sample is created by swapping pastry and blue fork and making plate as the head of pastry . We perform an extra validation step to filter out bad referring expressions. In this step, three additional AMT workers select a bounding box to identify the target object, and we only select the samples where at least two workers achieve IoU > 0.5 with the target object. Since the samples in Ref-Adv mainly differ in linguistic structure with respect to Ref-Hard , we hope that a model which does not make use of linguistic structure (and correspondingly spatial relations between objects) performs worse on Ref-Adv even when it performs well on Ref-Hard due to exploiting biases in the training data. Ref-Easy and Ref-Hard (Figure 6 in appendix) and consists of rich and diverse spatial relationships (Figure 7 in appendix). Concurrent to our work, Gardner et al. (2020) also propose perturbed test splits for several tasks by modifying in-domain examples. Ref-Adv expressions are longer on average than Ref-Easy and Ref-Hard (Figure 6 in appendix) and consists of rich and diverse spatial relationships (Figure 7 in appendix). In their setup, the original authors of each task create perturbed examples, whereas we use crowdworkers. Closest to our work is from Kaushik et al. (2020) who also use crowdworkers. While we use perturbed examples to evaluate robustness, they also use them to improve robustness (we propose complementary methods to improve robustness  5). Moreover, we are primarily concerned with the robustness of models for visual expression recognition task, while Gardner et al. and Kaushik et al. focus on different tasks (e.g., sentiment, natural language inference). 3.1 Human Performance on Ref-Easy , Ref-Hard and Ref-Adv We conducted an additional human study (on AMT) to compare the human performance on Ref-Easy , Ref-Hard and Ref-Adv splits. Concurrent to our work, Gardner et al. (2020) also propose perturbed test splits for several tasks by modifying in-domain examples. In their setup, the original authors of each task create perturbed examples, whereas we use crowdworkers. Closest to our work is from Kaushik et al. (2020) who also use crowdworkers. While we use perturbed examples to evaluate robustness, they also use them to improve robustness (we propose complementary methods to improve robustness  5). Moreover, we are primarily concerned with the robustness of models for visual expression recognition task, while Gardner et al. and Kaushik et al. focus on different tasks (e.g., sentiment, natural language inference). First, we randomly sampled 100 referring expressions from each of the three splits. Each referring expression is then assigned to three AMT workers and are asked to select a bounding box to identify the target object. We considered a sample to be correctly annotated by humans if at least two out of three workers select the ground-truth annotation. 3.1 Human Performance on Ref-Easy , Ref-Hard and Ref-Adv We conducted an additional human study (on AMT) to compare the human performance on Ref-Easy , Ref-Hard and Ref-Adv splits. First, we randomly sampled 100 referring expressions from each of the three splits. Each referring expression is then assigned to three AMT workers and are asked to select a bounding box to identify the target object. We considered a sample to be correctly annotated by humans if at least two out of three workers select the ground-truth annotation. evaluation, we obtained human performance on each of the three splits Ref-Easy, Ref-Hard, and Ref-Adv as 98%, 95%, and 96% respectively. 4 Diagnosing Referring Expression Recognition models We evaluate the following models, most of which are designed to exploit linguistic structure. Through this evaluation, we obtained human performance on each of the three splits Ref-Easy, Ref-Hard, and Ref-Adv as 98%, 95%, and 96% respectively. 2017; Andreas et al. 2016) grounds expressions using neural modules by decomposing an expression into < subject, relation, object > triples. The subject and object are localized to the objects in the image using a localization module while the relation between them is modeled using a relationship module. The full network learns to jointly decompose the input expression into a triple while also recognizing the target object. GroundNet (Cirik et al., 2018a) is similar to CMN, however it makes use of rich linguistic structure (and correspondingly rich modules) as defined by an external syntactic parser. CMN (Compositional Modular Networks; Hu et al. 2017; Andreas et al. 2016) grounds expressions using neural modules by decomposing an expression into < subject, relation, object > triples. The subject and object are localized to the objects in the image using a localization module while the relation between them is modeled using a relationship module. The full network learns to jointly decompose the input expression into a triple while also recognizing the target object. MattNet (Yu et al., 2018) generalizes CMN to flex-ibly adapt to expressions that cannot be captured by the fixed template of CMN. GroundNet (Cirik et al., 2018a) is similar to CMN, however it makes use of rich linguistic structure (and correspondingly rich modules) as defined by an external syntactic parser. It introduces new modules and also uses an attention mechanism to weigh modules. MattNet (Yu et al., 2018) generalizes CMN to flex-ibly adapt to expressions that cannot be captured by the fixed template of CMN. It introduces new modules and also uses an attention mechanism to weigh modules. ViLBERT (Lu et al., 2019), the state-of-the-art model for referring expression recognition, uses a Co-TRM TRM Co-TRM TRM Shared ViLBERT Layers Input image Input question/ Referring expression Task-Specific Layers Figure 3: Multi-task learning model for referring expression recognition with GQA pretrain-then-transfer learning approach to jointly learn visiolinguistic representations from large-scale data and utilizes them to ground expressions. This is the only model that does not explicitly model compositional structure of language, but BERT-like models are shown to capture syntactic structure latently (Hewitt and Manning, 2019). We trained on the full training set of RefCOCOg and performed hyperparameter tuning on a development set. We used the development and test splits of Mao et al. (2016). Table 2 shows the model accuracies on these splits and our proposed datasets. The models are trained to select ground truth bounding box from a set of predefined bounding boxes. We treat a prediction as positive if the predicted bounding box has IoU > 0.5 with the ground truth. Although the overall performance on the test set seem high, in reality, models excel only at Ref-Easy while performing poorly on Ref-Hard . The difference in performance between Ref-Easy and Ref-Hard ranges up to 15%. This indicates that current models do not exploit linguistic structure effectively. When tested on Ref-Adv , the performance goes down even further, increasing the gap between Ref-Easy and Ref-Adv (up to 26%). This suggests that models are relying on reasoning shortcuts found in training than actual understanding. Among the models, GroundNet performs worse, perhaps due to its reliance on rigid structure predicted by an external parser and the mismatches between the predicted structure and spatial relations between objects. ViLBERT achieves the highest performance and is relatively more robust than other models. In the next section, we propose methods to further increase the robustness of ViLBERT. We extend ViLBERT in two ways, one based on contrastive learning using negative samples, and the other based on multi-task learning on GQA (Hudson and Manning, 2019), a task that requires linguistic and spatial reasoning on images. Contrastive learning using negative samples Instead of learning from one single example, contrastive learning aims to learn from multiple examples by comparing one to the other. In order to increase the sensitivity to linguistic structure, we mine negative examples that are close to the current example and learn to jointly minimize the loss on the current (positive) example and maximize the loss on negative examples. We treat the triplets (cid:0) i, e, b (cid:1) in the training set as positive examples, where i , e , b stands for image, expression and ground truth bounding box. For each triplet (cid:0) i, e, b (cid:1) , we sample another training example (cid:0) i (cid:48) , e (cid:48) , b (cid:48) (cid:1) , and use it to create two negative samples, defined by (cid:0) i (cid:48) , e, b (cid:48) (cid:1) and (cid:0) i, e (cid:48) , b (cid:1) , i.e., we pair wrong bounding boxes with wrong expressions. For efficiency, we only consider negative pairs from the mini-batch. We modify the batch loss function as follows: L (cid:0) i , e , b (cid:1) = F ( e , e (cid:48) ) (cid:2) (cid:96) (cid:0) i , e , b (cid:1)  (cid:96) (cid:0) i , e (cid:48) , b (cid:1)   (cid:3) + + F ( i , i (cid:48) ) (cid:2) (cid:96) (cid:0) i , e , b (cid:1)  (cid:96) (cid:0) i (cid:48) , e , b (cid:48) (cid:1)   (cid:3) + Model Dev Test Easy Hard Adv ViLBERT (VB) 83.39 83.63 85.93 72.00 70.90 VB+ Sum-H 81.61 83.00 85.93 70.60 72.30 VB+ Max-H 82.93 82.70 86.58 70.46 73.35 VB+ MTL (GQA) 83.45 84.30 86.23 73.79 73.92 Table 3: Accuracy of enhanced ViLBERT models. Here (cid:96) ( i, e, b ) is the cross-entropy loss of ViLBERT, [ x ] + is the hinge loss defined by max (cid:0) 0 , x (cid:1) , and  is the margin parameter. F indicates a function over all batch samples. We define F to be either sum of hinges (Sum-H) or max of hinges (Max-H). While Sum-H takes sum over all negative samples, If batch size is n , for each (cid:0) i, e, b (cid:1) , there will be n  1 triplets of (cid:0) i (cid:48) , e, b (cid:48) (cid:1) and (cid:0) i, e (cid:48) , b (cid:1) . For (cid:0) i, e, b (cid:1) , there will be one (cid:0) i (cid:48) , e, b (cid:48) (cid:1) and one (cid:0) i, e (cid:48) , b (cid:1) . Similar proposals are known to increase the robustness of vision and language problems like visual-semantic embeddings and image description ranking (Kiros et al., 2014; Gella et al., 2017; Faghri et al., 2018). Multi-task Learning (MTL) with GQA In order to increase the sensitivity to linguistic structure, we rely on tasks that require reasoning on linguistic structure and learn to perform them alongside our task. We employ MTL with GQA (Hudson and Manning, 2019), a compositional visual question answering dataset. Specifically, we use the GQA-Rel split which contains questions that require reasoning on both linguistic structure and spatial relations (e.g., Is there a boy wearing a red hat standing next to yellow bus? as opposed to Is there a boy wearing hat? ). Figure 3 depicts the neural architecture. We share several layers between the tasks to enable the model to learn representations useful for both tasks. Each shared layer constitute a co-attention transformer block (Co-TRM; Lu et al. 2019) and a transformer block (TRM; Vaswani et al. 2017). While in a transformer, attention is computed using queries and keys from the same modality, in a co-attention transformer they come from different modalities (see cross arrows in Figure 3). The shared representations are eventually passed as input to task-specific MLPs. We optimize each task using alternative training (Luong et al., 2015). Results and discussion Table 3 shows the experimental results on the referring expression recognition task. Although contrastive learning improves e1 : The ladder that is raised the tallest e2 : A wooden boat carries 5 boys with skis e1' : The ladder in front of the raised ladder e2' : A pair of skis in the boat ViLBERT MTL GT Figure 4: Predictions of ViLBERT and MTL model (GT denotes ground-truth). the robustness of ViLBERT on Ref-Adv (+1.4% and +2.5% for Sum-H and Max-H respectively), it comes at a cost of slight performance drop on the full test (likely due to sacrificing biases shared between training and test sets). Whereas MTL improves the robustness on all sets showing that multitask learning helps (we observe 2.3% increase on GQA  A.5.2). Moreover, the performance of MTL on Ref-Hard and Ref-Adv are similar, suggesting that the model generalizes to unseen data distribution. Figure 4 shows qualitative examples comparing MTL predictions on Ref-Hard and Ref-Adv parallel examples. These suggest that the MTL model is sensitive to linguistic structure. However, there is still ample room for improvement indicated by the gap between Ref-Easy and Ref-Hard (12.4%). Our work shows that current datasets and models for visual referring expressions fail to make effective use of linguistic structure. Although our proposed models are slightly more robust than existing models, there is still significant scope for improvement. We hope that Ref-Hard and Ref-Adv will foster more research in this area. We would like to thank Volkan Cirik, Licheng Yu, Jiasen Lu for their help with GroundNet, MattNet and ViLBERT respectively, Keze Wang for his help with technical issues, and AWS AI data team for their help with Mechanical Turk. We are grateful to the anonymous reviewers for their useful feedback.
abstain result method method result objective other abstain abstain abstain abstain abstain abstain abstain abstain abstain method result method abstain result objective abstain abstain method abstain abstain abstain method method abstain method abstain abstain method abstain abstain method method method abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain method method abstain method method abstain method abstain abstain method method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain method abstain abstain abstain method method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain result objective abstain other other