We propose VALSE ( V ision A nd L anguage S tructured E valuation), a novel benchmark designed for testing general-purpose pretrained vision and language (V&L) models for their visio-linguistic grounding capabilities on specific linguistic phenomena . VALSE offers a suite of six tests covering various linguistic constructs. Solving these requires models to ground linguistic phenomena in the visual modality, allowing more fine-grained evaluations than hitherto possible. We build VALSE using methods that support the construction of valid foils, and report results from evaluating five widely-used V&L models. Our experiments suggest that current models have considerable difficulty addressing most phenomena. Hence, we expect VALSE to serve as an important benchmark to measure future progress of pretrained V&L models from a linguistic perspective , complementing the canonical task-centred V&L evaluations. General-purpose pretrained vision and language (V&L) models have gained notable performance on many V&L tasks (Lu et al., 2019; Tan and Bansal, 2019; Li et al., 2019; Chen et al., 2020; Li et al., 2020a; Su et al., 2020). As a result, V&L research has changed its focus from task-specific architectures to fine-tuning large V&L models. Current benchmarks give a good perspective on model performance on a wide range of V&L tasks (Cao et al., 2020; Lourie et al., 2021; Li et al., 2021), but the field is only starting to assess why models perform so well and whether models learn specific capabilities that span multiple V&L tasks . Specifically, we lack detailed understanding of the extent to which such models are able to ground linguistic phenomenafrom morphosyn-tax to semanticsin the visual modality (Bernardi  Corresponding author parcalabescu@cl. uni-heidelberg.de . and Pezzelle, 2021). For example, recent evidence suggests that models are insensitive to linguistic distinctions of verb-argument structure (Hendricks and Nematzadeh, 2021) and word order (Cirik et al., 2018; Akula et al., 2020). Our work addresses this gap with VALSE (Vi-sion And Language Structured Evaluation), a benchmark for V&L model evaluation comprising six tasks, or pieces', where each piece has the same structure: given a visual input, a model is asked to distinguish real captions from foils , where a foil is constructed from a caption by altering a word or phrase that realizes a specific linguistic phenomenon , e.g., semantic number of nouns, verb argument structure, or coreference. VALSE uses a resource-lean diagnostic setup that dispenses with large-scale annotation (e.g., of bounding boxes), and builds on existing high-quality image captioning and VQA data. VALSE is designed to leverage the existing prediction heads in pretrained (or finetuned) V&L models; for that reason, our benchmark does not include any re-training and can be interpreted as a zero-shot evaluation. We build test data for each piece so as to safeguard against the possibility of models exploiting artefacts or statistical biases in the data, a well-known issue with highly parameterised neural models pretrained on large amounts of data (Goyal et al., 2017; Madhyastha et al., 2018; Kafle et al., 2019). With this in view, we propose novel methods to guard against the emergence of artefacts during foiling. Our main contributions are: i) We introduce VALSE, a novel benchmark aimed at gauging the sensitivity of pre-trained V&L models to foiled instances. ii) We cover a wide spectrum of basic linguistic phenomena affecting the linguistic and visual modalities: existence, plurality, counting, spatial relations, actions, and entity coreference. iii) We investigate novel strategies to build valid foils that include automatic and human valida-8253 tion. We balance word frequency distributions between captions and foils, and test against pretrained models solving the benchmark uni-modally by relying only on text. We employ masked language modeling (MLM) in foil creation and semantic inference for validating foils, and finally collect human annotations for the entire benchmark. iv) We establish initial experimental results for pretrained V&L models of diverse architectures on VALSE. The overall weak performance of these models indicates that the time is ripe for a novel, reliable foiling dataset targeting the visual grounding capabilities of V&L models through the lens of linguistic constructs. 1 2 Background and Related work Pretrained V&L models learn to combine vision and language through self-supervised multitask learning. Tasks include multimodal masked modeling where words in the text and object labels or regions in the image are masked out, then predicted and image-sentence alignment , whereby a model learns to predict whether an image and a text correspond. Major architectures are singleand dual-stream multimodal transformers: single-stream models concatenate word and image features, and encode the resulting sequence with a single transformer stack; dual-stream models use distinct transformer stacks to handle visual and textual inputs, and additional layers (e.g. co-attention) to fuse these into multimodal features. Benchmarking V&L models V&L models (Li et al., 2019; Lu et al., 2019; Tan and Bansal, 2019; Lu et al., 2020; Li et al., 2020b; Kim et al., 2021) are commonly evaluated on V&L tasks such as VQA (Goyal et al., 2017), visual reasoning (Suhr et al., 2019), or image retrieval (Lin et al., 2014; Plummer et al., 2015). Given how well transformer-based models perform across unimodal and multimodal tasks, research efforts have recently started to address what makes them so effective, and to what extent they learn generalisable representations. Techniques to address these questions in unimodal and multimodal V&L contexts include: adversarial examples (Jia and Liang, 2017; Jia et al., 2019); investigation 1 We release our dataset containing all annotators' votes (Prabhakaran et al., 2021) at https://github.com/ Heidelberg-NLP/VALSE . of the impact of bias, be it linguistic (Gururan-gan et al., 2018), visual semantic (Agarwal et al., 2020), or socio-economic (Garg et al., 2019); and the use of linguistically-informed counterfactual and minimally-edited examples (Levesque et al., 2012; Gardner et al., 2020). A trend within the latter research line that is specific to V&L models is vision-and-language foiling (Shekhar et al., 2017b; Gokhale et al., 2020; Bitton et al., 2021; Parcalabescu et al., 2021; Rosenberg et al., 2021), where the idea is to create counterfactual (i.e., foiled ) and/or minimally edited examples by performing data augmentation on captions (Shekhar et al., 2017b,a) or images (Rosenberg et al., 2021). Since most V&L models are pretrained on some version of the image-text alignment task, it is possible to test their ability to distinguish correct from foiled captions (in relation to an image) in a zero-shot setting. The construction of foils can serve many investigation purposes. With VALSE, we target the linguistic grounding capabilities of V&L models , focusing on pervasive linguistic phenomena that span multiple tokens , described in 3.1 3.6. At the same time, we ensure that our data is robust to perturbations and artefacts by i) controlling for word frequency biases between captions and foils, and ii) testing against unimodal collapse , a known issue of V&L models (Goyal et al., 2017; Madhyastha et al., 2018), thereby preventing models from solving the task using a single input modality. The issue of neural models exploiting data artefacts is well-known (Gururangan et al., 2018; Jia et al., 2019; Wang et al., 2020b; He et al., 2021) and methods have been proposed to uncover such effects, including gradient-based, adversarial perturbations or input reduction techniques (cf. Wallace et al., 2020). Yet, these methods are still not fully understood (He et al., 2021) and can be unreliable (Wang et al., 2020b). Our work is related to Gardner et al. (2020), who construct task-specific contrast sets for NLU. However, our focus is on modelling linguistic phenomena instead of tasks, and we construct carefully curated, balanced, single foils from valid instances that we select from multiple multimodal datasets. We resort to a musical analogy to describe VALSE: Vision And Language Structured Evaluation is composed of 6 pieces , each corresponding to a specific linguistic phenomenon (see Table 1 for an overview). Each piece consists of one or more instruments designed to evaluate a model's ability to ground that specific linguistic phenomenon. All instruments are built by applying foiling functions (FFs) specific to the linguistic phenomenon under study. FFs take a correct caption as input and change a specific part to produce a foiled caption (or foil ). We design FFs such that the sentences they produce fail to describe the image, while still being grammatical and otherwise valid sentences. Of course, a foiled caption may be less likely than the original caption from which it was produced, and such unwarranted biases can be easily picked up by overparameterised V&L models. Moreover, an automatic FF may fail to produce a foil that contradicts the image, for example by altering the original caption to yield a near-synonymous one, or one that is entailed by the original caption. For phenomena that make it difficult to control these crucial properties of foils, we apply additional filters: i) some FFs make use of strong LMs to propose changes to captions, so that the generated foils are still high-probability sentences; ii) we use state-of-the-art natural language inference (NLI) methods to detect cases where there is an entailment between caption and foil, and filter out such foils from the dataset (see 4 for discussion). As a final measure, we employ human annotators to validate all generated testing data in VALSE. datasets. Below, we describe each piece and its instruments, and the corresponding task setup in VALSE. For each instrument, we follow the same procedure: i) we identify captions that contain instances of the targeted linguistic phenomenon; ii) we apply a FF that automatically replaces the expression with a variant that contradicts the original expression's visual content, thereby constructing one or more foils from each target instance in the original caption, as discussed in 4; we then iii) subject the obtained foils to various filters, with the aim of distilling a subset of valid and reliable foils that cannot be easily tricked by a new generation of highly parameterised pretrained V&L models. The existence piece has a single instrument and targets instances with existential quantifiers . Models need to differentiate between examples i) where there is no entity of a certain type or ii) where one or more of these entities are visible in an image. We use the Visual7W visual question answering dataset (Zhu et al., 2016) and source its how many' examples, building a pool of those whose answers are numerals (0, 1, 2, etc.). We use templates to transform question and answer fields into a declarative statement that correctly describes what can be seen in the image, e.g. Q: How many animals are shown? A: 0'  There are 0 animals shown'. We then transform these statements into an existential 8255 statement. In the example above, we replace the numeral by the word no' to create a correct caption (There are no animals shown') and remove the numeral altogether to create a foil (There are animals shown'). The existence piece has 505 image captionfoil tuples after manual validation, out of 534 candidates (cf. 4), and captions/foils are balanced: 50% of the (correct) captions originally have answer 0, and the remaining have answer 1 or greater. Full details are provided in A.1. The plurality piece has a single instrument, concerned with semantic number . It is intended to test whether a model is able to distinguish between noun phrases denoting a single entity in an image (exactly one flower'), versus multiple entities (some flowers'). The dataset consists of 851 validated instances out of 1000 generated candidates (cf. 4), evenly divided between cases where the caption contains a plural NP, foiled by replacing it with a singular ( pl2sg : some flowers'  exactly one flower'), or conversely, the caption contains a singular which is foiled by replacing it with a plural ( sg2pl ). Foil candidates were generated from the COCO 2017 validation set (Chen et al., 2015). Full details about the foil construction and our measures against introducing biases with quantifiers such as exactly one', are provided in A.2. The counting piece has three instruments: balanced , adversarial and small numbers . All instances are statements about the number of entities visible in an image . The model needs to differentiate between examples where the specific number of entities in the associated image is correct or incorrect, given the statement. Similarly to the existence piece, we use the Visual7W VQA dataset (Zhu et al., 2016) and source its how many' examples whose answers are numerals (0, 1, 2, etc.). We use templates to transform question and answer fields into a declarative statement describing the image and create foils by replacing the numeral in the correct statement by another numeral. All three instruments are designed to show whether models learn strategies that generalize beyond the training distribution, and to what extent a model exploits class frequency bias. 2 In counting balanced we cap the number of examples to 2 We take the original answer in Visual7W as the example class: e.g., in There are 0 animals shown', the class is 0. a maximum per class and make sure correct and foil classes are balanced, so that models that exploit class frequency bias are penalized. In counting adversarial we ensure that all foils take class n  { 0 , 1 , 2 , 3 } , whereas all correct captions take class m  { m | m  4 } . Biased models are expected to favour more frequent classes. Since small numbers are naturally the most frequent, models that resort to such biases should perform poorly on this adversarial test set. Counting small numbers is a sanity check where all correct captions and foils have class n  { 0 , 1 , 2 , 3 } , and caption/foil classes are balanced. Since models likely have been exposed to many examples in this class set and all such classes are high-frequency, with this instrument we disentangle model performance from class exposure. Counting balanced, adversarial, and small numbers have 868 (1000), 691 (756), and 900 (1000) instances after (before) manual validation, respectively (cf. 4). For details, see A.3. The relations piece has a single instrument and focuses on the ability of models to distinguish between different spatial relations. Foils differ from the original caption only by the replacement of a spatial preposition. As with plurals, the data was sourced from the COCO 2017 validation split. To create foils, we first identified all preposition sequences in captions (e.g., in', out of'). Foils were created by masking the prepositions and using SpanBERT (Joshi et al., 2020) to generate candidates of between 13 words in length. We keep SpanBERT candidates, which are spans whose lengths vary from 1 to 3, if they differ from the original preposition sequence, but exist in the dataset. There are 535 instances after manual validation out of 614 proposed instances (cf. 4), and we ensure that prepositions are similarly distributed among captions and foils. Full details are provided in A.4. The actions piece has two instruments: i) action replacement and ii) actant swap . They test a V&L model's capability to i) identify whether an action mentioned in the text matches the action seen in the image (e.g., a man shouts / smiles at a woman'), and ii) correctly identify the participants of an action and the roles they play (e.g., is it the man who is shouting or is it the woman, given the picture in Table 1?). 504 action verbs, and we generate captions and foils from SWiG annotations of semantic roles and their fillers. For the action replacement piece, we exchange action verbs with other verbs from SWiG that fit the linguistic context as suggested by BERT. For the actant swap, we swap role fillers in the role annotations, hence generating action descriptions with inverted roles. Action replacement and actant swap have 648 (779) and 949 (1042) instances after (before) manual validation, respectively (cf. 4). See A.5 for full details. The coreference piece aims to uncover whether V&L models are able to perform pronominal coreference resolution. It encompasses cases where i) the pronoun has a noun (phrase) antecedent and pronoun and (noun) phrase are both grounded in the visual modality (A woman is driving a motorcycle. Is she wearing a helmet?'), and cases where ii) the pronoun refers to a region in the image or even to the entire image (Is this outside?'). We create foils based on VisDial v1.0 (Das et al., 2017) with images from MSCOCO (Lin et al., 2014). VisDial captions and dialogues are Q&A sequences. We select image descriptions of the form [ Caption. Question? Yes/No. ] where the question contains at least one pronoun. When foiling, we exchange the answer from yes to no and vice-versa (see Table 1). We ensure a 50-50% balance between yes / no answers. The coreference piece consists of two instruments: coreference standard originating from the VisDial train set and a small coreference clean set from the validation set, containing 708 (916) and 104 (141) examples after (before) manual validation, respectively (cf. 4). 3 See A.6 for full details. In VALSE, an instance consisting of an image-caption-foil triple is considered valid if: the foil minimally differs from the original caption; the foil does not accurately describe the image; and independent judges agree that the caption, but not the foil, is an accurate description of the image. We consider a foiling method to be more reliable the more it ensures that a generated foil does not substantially differ from a human caption regarding distributional and plausibility bias, and cannot be easily solved unimodally. In this section, we discuss automatic and manual means to reliably construct valid foils. In this context, two types of bias are especially worthy of note: distributional bias (4.1) and plausibility bias (4.2). In 4.3 we discuss how we apply a natural language inference model to filter examples in our data pipeline, and 4.4 show how we manually validate all examples in our benchmark. Random samples from the final version of each instrument are shown in Tab. 611. A first form of bias is related to distributional imbalance between captions and foils (e.g., certain words or phrases having a high probability only in foils). Previous foiling datasets exhibit such imbalance, enabling models to solve the task disregarding the image (Madhyastha et al., 2019). To mitigate this problem, for each phenomenon and throughout our data creation process, we ensure that the token frequency distributions in correct and foiled captions are approximately the same (cf. App. A and E). A second form of bias may arise from automatic procedures yielding foils that are implausible or unnatural, which can facilitate their detection. Often, VALSE pieces can be safely foiled by simple rules (e.g., switching from existence to non-existence, or from singular to plural or vice versa). However, with spatial relations and actions , a foil could be deemed unlikely given only the textual modality and independently of the image, e.g., a man stands under / on a chair'. Such plausibility biases may be detected by large language models that incorporate commonsense knowledge (Petroni et al., 2019; Wang et al., 2020a), and we expect future V&L models to exhibit similar capabilities. To ensure that foiled and correct captions are similarly plausible, we use language models such as BERT (Devlin et al., 2019) and SpanBERT (Joshi et al., 2020) to suggest replacements in our foiling functions. Additionally, in the case of spatial relations and plurals, we also apply a grammaticality filter using GRUEN (Zhu and Bhat, 2020). GRUEN was originally proposed to automatically score generated sentences based on discourse-level and grammatical properties. We use only the grammaticality component of GRUEN, and retain only foil candidates with a grammaticality score  0 . 8 . benchmark could be solved by a multimodal model with strong linguistic capacities in unimodal collapse , whereby a model silently relies on a single modality within which biases are easier to exploit (Goyal et al., 2017; Shekhar et al., 2019a). By evaluating VALSE with unimodal models, we establish a baseline that V&L models should exceed if we are to expect true multimodal integration. When constructing foils, we need to ensure that they fail to describe the image. To test this automatically, we apply natural language inference (NLI) with the following rationale: We consider an image and its caption as a premise and its entailed hypothesis, respectively (a similar rationale is applied in the visual entailment task; Xie et al., 2019). In addition, we consider the caption as premise and the foil as its hypothesis . If a NLI model predicts the foil to be entailed (E) by the caption, it cannot be a good foil since by transitivity it will give a truthful description of the image. By contrast, if the foil is predicted to contradict (C) or to be neutral (N) with respect to the caption, we take this as an indicator of a valid (C) or a plausible (N) foil. 4 We use the NLI model ALBERT (Lan et al., 2020) finetuned on the task (see Appendix C for details). Filtering with NLI was initially applied to relations, plurals and actions , on the grounds that foils in these pieces may induce substantive changes to lexical content. 5 Following automatic labelling of caption-foil pairs, we manually validated a sample labelled as E, C or N. For relations ( N = 30 ), labels were found to be near 100% accurate with only 2 (0.06%) errors overall. For plurals ( N = 60 , 50% sg2pl and 50% pl2sg ), the error rate was also low, with 0 errors for C, 33% errors for E and 11% errors for N. Here, a number of entailment errors were due to odd formulations arising from the automatic foiling process, whereas no such oddities were observed for C. We therefore include only foils labelled C in the final relations and plurals pieces. For actions , the model labelled 4 See the following examples from action replacement: P: A mother scolds her son. H1: A mother encourages her son. (C; good foil); H2: A mother camps with her son. (N; needs image control); H3: A mother talks to her son. (E; not a suitable foil) If the NLI prediction is N, we still need to check the image, since the description might happen to fit the image content. 5 By contrast, existence and counting foils involve a more straightforward swap (e.g., between numerical quantities); similarly, coreference foils simply involve the replacement of a positive with a negative answer. contradictions very accurately (0% error) but was erroneous up to 97.1% for E, meaning that a large number of valid foils would be spuriously excluded. To avoid reducing the dataset too much, we did not use NLI filtering for actions, but relied on human annotation as a final validity check. As a final step, the data for each instrument was submitted to a manual validation. For each instance, annotators were shown the image, the caption and the foil. Caption and foil were numbered and displayed above each other to make differences more apparent, with differing elements highlighted in boldface (Fig. 2, App. E). Annotators were not informed which text was the caption and which was the foil, and captions appeared first (numbered 1 ) 50% of the time. The task was to determine which of the two texts accurately described what could be seen in the image. In each case, annotators had a forced choice between five options: a) the first, but not the second; b) the second, but not the first; c) both of them; d) neither of the two; and e) I cannot tell. Each item was annotated by three individuals. The validation was conducted on Amazon Mechanical Turk with a fixed set of annotators who had qualified for the task. For details see App. E. For the final version of VALSE, we include instances which passed the following validation test: at least two out of three annotators identified the caption, but not the foil, as the text which accurately describes the image. Across all instruments, 87.7% of the instances satisfied this criterion (min 77.3%; max 94.6%), with 73.6% of instances overall having a unanimous (3/3) decision that the caption, but not the foil, was an accurate description. We consider these figures high, suggesting that the automatic construction and filtering procedures yield foils which are likely to be valid, in the sense discussed in 4 above. We compute inter-annotator agreement for each instrument (Tab. 5). On the valid subset, agreement is low to medium (Krippendorff's  : min=0.23, max=0.64, mean=0.42, sd=0.12). We note that there is considerable variation in the number of annotations made by individuals, and  is computed over 5 categories. Hence, this result cannot be straightforwardly interpreted as a ceiling of human performance for VALSE. However,  is higher for pieces on which models also perform better (e.g. 8258 existence, Foil-It!; cf. 5). We propose VALSE as a task-independent, zero-shot benchmark to assess the extent to which models learn to ground specific linguistic phenomena as a consequence of their pretraining (or fine-tuning). VALSE is built in the spirit of approaches such as Checklist (Ribeiro et al., 2020), including pairs consisting of captions and minimally edited foils. The only requirement to evaluate a model on our benchmark is: i) to have a binary classification head to predict whether an image-sentence pair is foiled, or ii) to predict an image-sentence matching score between the image and the caption vs. the foil, returning the pair with the highest score. Systems reporting results on VALSE are expected to report any data used in model training prior to testing on VALSE, for comparability. We employ five metrics 6 for evaluation: overall accuracy ( acc ) on all classes (foil and cor-rect); precision ( p c ) measuring how well models identify the correct examples; foil precision ( p f ) measuring how well foiled cases are identified; pairwise ranking accuracy ( acc r ), which measures whether the image-sentence alignment score is greater for a correct image-text pair than for its foiled pair; and area under the receiver operating characteristic curve (AUROC), which measures how well models distinguish correct vs. foiled examples across different prediction thresholds. acc r is more permissive than acc as it accepts model predictions if the score for a foil is lower than the caption's score. Our main metrics are AUROC and acc r . acc r gives results for a pair (cid:104) image, caption (cid:105) and (cid:104) image, foil (cid:105) . Both AUROC and acc r are well suited to evaluate minimally-edited pairs as neither uses a classification threshold. As for p c and p f , since these are competing metrics where naively increasing one can decrease the other, we report the smaller of the two as an indicator of how informed model predictions are. Since all instruments are implemented as a balanced binary classification, the random baseline is always 50%. We benchmark five V&L models on VALSE: CLIP (Radford et al., 2021), LXMERT (Tan and Bansal, 6 All metrics are defined in Appendix B. Multimodal results The best zero-shot results are achieved by ViLBERT 12-in-1 with the highest scores across the board, followed by ViLBERT, 8259 Metric Model Existence Plurality Counting Sp.rel. 2019), ViLBERT (Lu et al., 2019), ViLBERT 12-in-1 (Lu et al., 2020), and VisualBERT (Li et al., 2019). These models have different architectures and are pretrained on a variety of tasks with different training data. We also benchmark two unimodal text-only models, GPT1 (Radford et al., 2018) and GPT2 (Radford et al., 2019). See Appendix D for details on all these models used in our evaluation. Unimodal models GPT1 and GPT2 are autoregressive language models pretrained on English text. We test whether VALSE is solvable by these unimodal models by computing the perplexity of the correct and foiled caption and predicting the entry with the lowest perplexity . If the perplexity is higher for the foil, we take this as an indication that the foiled caption may suffer from plausibility bias or other linguistic biases (cf. 4.2). We test V&L and unimodal models on VALSE in a zero-shot setting, and also evaluate on a number of correct captions and foils from the FOIL it! dataset (Shekhar et al., 2017b) (cf. App. A.7 for details). All results are listed in Table 2. Unimodal results For most instruments, unimodal results are close to random and hence do not signal strong linguistic or plausibility biases. One exception is the original FOIL it! dataset, in line with Madhyastha et al. (2019)'s findings. Also the spatial relations (77.2%), action replacement (66.8%) and actant swap (76.9%) instruments suggest plausibility biases in foils. Such biases are hard to avoid in automatic foil generation for actions due to the verb arguments' selectional restrictions, which are easily violated when flipping role fillers, or replacing the verb. Similar considerations hold for relations: though SpanBERT proposals are intended to aid selection of likely replacements for prepositions, plausibility issues arise with relatively rare argument-preposition combinations. While these might be the first instruments in VALSE to be solved in the future, current V&L models struggle to detect even blatant mismatches of actant swap, e.g., A ball throws a tennis player.' For VALSE, the unimodal scores will serve as a baseline for the pairwise accuracy of V&L models. LXMERT, CLIP, 7 and finally VisualBERT. The latter obtains high p f but very low p c values reflected in the min( p c , p f ) scoresindicating that VisualBERT learned a heuristic that does not generalise (see Hendricks and Nematzadeh, 2021, for similar observations with other models). We hypothesise that this is due to the way image-sentence alignment is framed in VisualBERT's pretraining: the model expects an image and a correct sentence c 1 , and predicts whether a second sentence c 2 is a match. 8 During pretraining c 1 and c 2 are likely to differ in many ways, whereas in our setting, they are nearly identical. This may bias the model against predicting foils, which would raise the value p f . Instruments centered on individual objects like existence and the FOIL it! dataset are almost solved by ViLBERT 12-in-1, highlighting that models are capable of identifying named objects and their presence in images. However, none of the remaining pieces can be reliably solved in our adversarial foiling settings: i) distinguishing references to single vs. multiple objects or counting them in an 7 CLIP works in a contrastive fashion, therefore we report only acc r (cf. Appendix D for details). 8 c 1 is one of the 5 captions describing the relevant image in MSCOCO. During VisualBERT's pretraining, c 2 can be an alternative caption out of these 5, or a randomly drawn caption which does not describe the image. The pretraining task is to determine if c 2 correctly describes the image or not. image (plurality and counting); ii) correctly classifying a named spatial relation between objects in an image (relations); iii) distinguishing actions and identifying their participants, even if supported by preference biases (actions); or, iv) tracing multiple references to the same object in an image through the use of pronouns (coreference). Correct vs. foil precision p c and p f show that V&L models struggle to solve the phenomena in VALSE. When a model achieves high precision on correct captions p c this is often at the expense of very low precision on foiled captions p f (cf. ViL-BERT), or vice-versa (cf. VisualBERT). This suggests that such models are insensitive to VALSE's inputs: models that almost always predict a match will inflate p f at the expense of p c . min( p c , p f ) reveals that VisualBERT and ViLBERT perform poorly and below random baseline, and LXMERT close to or below it. ViLBERT 12-in-1 performs strongly on existence, well on counting, but struggles on plurality, spatial relations, coreference, and actions. These tendencies we see reflected in our main metrics, acc r and AUROC. We present the VALSE benchmark to help the community improve V&L models by hard-testing their visual grounding capabilities through the lens of lin-8260 guistic constructs. Our experiments show that V&L models identify named objects and their presence in images well (as shown by the existence piece), but struggle to ground their interdependence and relationships in visual scenes when forced to respect linguistic indicators. We encourage the community to use VALSE for measuring progress towards V&L models capable of true language grounding. Furthermore, VALSE could be used as an indirect assessment of datasets, as models could be evaluated before and after training or fine-tuning to see if a dataset helps models improve on any of the aspects tested by VALSE. VALSE is designed as a living benchmark. As future work we plan to extend it to further linguistic phenomena, and to source data from diverse V&L datasets to cover more linguistic variability and image distributions. IC has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skodowska-Curie grant agreement No 838188. AG and MC are supported by the European Union's Horizon 2020 research and innovation Programme under the Marie Skodowska-Curie grant agreement No 860621. This collaboration was facilitated by the Multi3Generation COST Action CA18231.
objective abstain abstain result abstain method abstain abstain abstain abstain abstain method abstain method method objective objective objective abstain objective objective method result objective objective abstain abstain other abstain other abstain other abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain abstain method method method abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain objective method abstain result abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain method abstain abstain result method method method abstain objective result abstain result result abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain method abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain result method abstain abstain objective other other other