We introduce CARETS, a systematic test suite to measure consistency and robustness of modern VQA models through a series of six fine-grained capability tests. In contrast to existing VQA test sets, CARETS features balanced question generation to create pairs of instances to test models, with each pair focusing on a specific capability such as rephrasing, logical symmetry or image obfuscation. We evaluate six modern VQA systems on CARETS and identify several actionable weaknesses in model comprehension, especially with concepts such as negation, disjunction, or hypernym invariance. Interestingly, even the most sophisticated models are sensitive to aspects such as swapping the order of terms in a conjunction or changing the number of answer choices mentioned in the question. We release CARETS to be used as an extensible tool for evaluating multi-modal model robustness. 1 1 Introduction The task of visual question answering integrates the domains of computer vision and NLP by probing models' understanding of images through natural language queries. After the introduction of the Visual Question Answering (VQA) benchmark (Antol et al., 2015), subsequent work identified the presence of several superficial correlations and other weaknesses latent in the VQA question gathering process (Goyal et al., 2017; Agrawal et al., 2018), which lead to potentially optimistic evaluations when considering accuracy alone. More recently developed benchmarks (Hudson and Manning, 2019; Goyal et al., 2017; Agrawal et al., 2018) explicitly avoid these weaknesses by introducing question, answer, and image balancing, or distributional shifts. While these 1 Source code, data, and additional resources may be found at https://github.com/princeton-nlp/CARETS efforts provide more difficult benchmarks, a thor-ough evaluation of model capabilities requires a deeper and more detailed approach. To this end, we introduce CARETS  a Consistency And Robustness Evaluative Test Suite for visual question answering. Inspired by recent work in NLP that generates unit tests' for models (Ribeiro et al., 2020b), CARETS contains systematically generated VQA tests that evaluate six different capabilities that any VQA model should handle  robustness to question rephrasings, ontological reasoning, symmetrical logic, visual perturbations, question negation, and attribute antonymy. Each test point in CARETS consists of a pair of instances which are small but strategic variations of each other either visually or in the question's text. This allows us to conduct fine-grained capability evaluations beyond just measuring high-level accuracy scores. Across tests, we generate over 190,000 instance pairs in total using a programmatic approach that fills in templates (from nearly 200 templates in total) using ground-truth scene graphs (Krishna et al., 2017) from the GQA (Hudson and Manning, 2019) validation split. We then evaluate six modern VQA models on each test using several metrics: overall accuracy, self-consistency and comprehensive accuracy . Self-consistency measures models' ability to maintain their answer across question variations, while comprehensive accuracy estimates their ability to answer all instance variants correctly. Our experiments reveal several interesting findings: (1) most modern VQA systems achieve only middling self-consistency (  60-80%) which is further not always correlated with their accuracy, (2) all models struggle to comprehend the concept of negation (self-consistency of 18-28% and comprehensive accuracy < 17%), and (3) even simple perturbations like changing the order of choices in the question text can induce a substantial drop 6392 Figure 1: Our consistency and robustness test suite (CARETS) consists of six tests, corresponding to six identified phenomena that VQA models should be robust to. (10-15%) in performance. Moreover, even state-of-the-art models like LXMERT (Tan and Bansal, 2020) are highly sensitive to the type of questions (binary vs multi-choice) and the number of choices provided. These results reveal several shortcomings in modern VQA systems and hint at potential areas for future improvement. Going beyond our current discoveries, CARETS is an extensible framework and may be easily extended by adding new capability tests for fine-grained evaluation of future models. VQA evaluation. The textual, visual, and answer biases discovered in the VQA dataset (Antol et al., 2015) spurred on recent work seeking to improve model evaluation for the task by eliminating these biases (Goyal et al., 2017), introducing distributional shifts (Agrawal et al., 2018), requiring model explanations (Li et al., 2018), thoroughly analyzing biases in datasets and models (Man-junatha et al., 2019), or evaluating on different recognition subtasks (Kafle and Kanan, 2017). While debiased and challenging benchmarks are important, their focus on accuracy as the sole evaluation metric leaves much to be desired (Ribeiro et al., 2020a; Kervadec et al., 2020). In contrast, our testbed provides question or image pairs that compares models' predictions between questions; measuring their accuracy, consistency, and robustness to a variety of text and image perturbations. Synthetic Dataset Generation for VQA. One way in which we can generate diverse and balanced datasets is to generate them synthetically, as is done by (Johnson et al., 2015; Zhang et al., 2016; Hudson and Manning, 2019). Synthetically generating questions, images, or both, allows fine control over the distribution of questions, answers, and images. Additionally for our case, synthetic generation allows us to control not just the particular semantics of one question, but also how one question relates to another question in a precisely defined way (e.g. one question is a negation of another) while also remaining relevant and grounded in the image. As both the CLEVR (Johnson et al., 2015) and GQA (Hudson and Manning, 2019) datasets use image scene graphs for question and label generation, they contain questions combining a variety of required capabilities, including compositionality. We feature real-world images with synthetically generated questions as well, but in contrast to GQA, our evaluation has instance pairs to systematically test a focused set of capabilities, showing that models may still struggle with simpler, non-compositional questions. Consistency as Model Comprehension. Some recent work has sought to evaluate models using consistency and other metrics (Hudson and Manning, 2019; Shah et al., 2019; Ribeiro et al., 2020a; Selvaraju et al., 2020; Bitton et al., 2021; Mouselinos et al., 2022). These evaluations of-6393 ten evaluate consistency through question entailment and implication, or simply contrasting examples in the case of (Bitton et al., 2021). The concurrent work (Mouselinos et al., 2022) takes a unique approach, performing discrete visual perturbations while preserving the semantic integrity of a question; their work complements our own as it focuses exclusively on visual perturbations in a synthetic image setting. While we consider these previous methods important for evaluating model comprehension, they often combine question types and capabilities, changing the kind of expected answer, or evaluating consistency on a tree or set of entailed questions. Though ideally models would be consistent and robust for these more complex types of tests, our approach reveals that models can fail even on simple implications. Our goal is to provide a testbed for fine-grained evaluation of VQA models' capabilities. To do so, we generate multiple tests, each corresponding to a specific model capability. In contrast to standard VQA benchmarks (Antol et al., 2015; Goyal et al., 2017; Hudson and Manning, 2019), our test sets consist of a pair of original and perturbed instances (cid:104) ( I 1 , q 1 , a 1 ) , ( I 2 , q 2 , a 2 ) (cid:105) , each with an image, a question and an answer. The two instances within a pair differ from each other in a minimal yet carefully constructed way to hone in on a particular capability, similar to BLiMP (Warstadt et al., 2020). A model is then evaluated on its overall accuracy , self-consistency (ability to produce consistent, even if wrong, answers to both instances within a pair), and comprehensive accuracy (ability to answer consistently and correctly for an instance pair). Overall, we create six tests corresponding to key capabilities. We provide a high-level description of each test here and describe generation details in Section 4. First, we create four invariance tests 2 that use variations of the question phrasing and expect the model to produce the same answer to both questions within an instance pair: 1. Rephrasing invariance ( REPHRASE-INV ) measures the model's understanding of mi-nor, meaning-preserving textual changes, e.g.: What color is the bottle on the shelf, white or blue? and  Does the color of the bottle on the shelf seem more white or blue? 2 Reusing terminology from Ribeiro et al. (2020b). 2. Ontological invariance ( ONTOLOGICALINV ) measures understanding of ontology, e.g. changing a hyponym in: Do you see a green jacket? to a hypernym Do you see any green clothes ? 3. Order invariance ( ORDER-INV ) measures understanding of logically equivalent questions containing different argument orders, e.g.: Is the black vehicle a van or a truck? and Is the black vehicle a truck or a van ? 4. Visual obfuscation invariance ( VISUALINV ) measures the model's answering ability when parts of the image not directly relevant to the visual question are removed. Specifi-cally, we explore blurring, masking and cropping techniques to modify the image. 5. Attribute antonym directional expectation ( ANTONYM-DIR ) measures the model's understanding of antonyms, e.g., Do you think that the wood table is short? and Do you think that the wood table is long ? 6. Negation directional expectation ( NEGATION-DIR ) measures a model's grasp of negation, e.g.,  Are there any apples in this picture? and Are there no apples in this picture? Each of the six test datasets start with the generation of original', unperturbed instances ( I 1 , q 1 , a 1 ) (Section 4.1). Then, for each such instance, we generate a variation ( I 2 , q 2 , a 2 ) by either perturbing the original question q 1 or image I 1 (Section 4.2). Further, each test set is composed of a diverse set of questions. These may broadly be grouped into verification (or binary ) questions, with expected answers being either yes or no , and multi-choice questions, with expected answers derived from a list of objects or attributes provided in the question. Questions for each test are generated from question templates (examples for each are provided in Appendix A.1) grouped into the following types. Q1: Object verification (54 templates): e.g., Is Q2: Conjunctive verification (18 templates): e.g., Is there a <obj1> and a <obj2> in the image? Q3: Disjunctive verification (18 templates): e.g., Is there a <obj1> or a <obj2> in the image? Q4: Attribute verification (25 templates): e.g., Is the <obj> in the image <attr> ? Q5: Object multi-choice (25 templates): e.g., Is the <obj-class> , <choices> ? Q6: Attribute multi-choice (39 templates): e.g., What sort of <attr-class> is the <obj> , <choices> ? Q7: Action multi-choice (28 templates): e.g., What is the <action-class> that the <obj> doing, <choices> ? Words in typewriter font represent template arguments. Generally, each <obj> argument can be filled by a singular object (cup) or an at-tribute+object (red cup) while <attr> arguments are filled with singular attributes (shiny). For object verification (Q1), attribute verification (Q4), attribute multi-choice (Q6), and action multi-choice (Q7) questions, some templates let <obj> arguments be filled with an object related to another object (e.g. red cup on the table); this type is excluded from conjunctive verification (Q2) and disjunctive verification (Q3) questions to prevent the generation of verbose questions. For the multi-choice questions (Q5, Q6, Q7), <choices> are replaced with a list of 2 or 3 singular objects, attributes, or actions respectively (e.g. cow or horse or wood, metal, or plas-tic). The <obj-class> argument is filled with a hypernym of all object choices and always appears with either an attribute or a related object (black animal, animal eating grass). The <attr-class> argument is filled with the attribute category of all attribute choices (e.g. ma-terial). Finally, the <action-class> argument is filled with the action category of all action choices (e.g. sport). Question argument generation. The question arguments are generated using images from the validation split of the GQA dataset (Hudson and Manning, 2019) which contains 10,696 images manually annotated with 1,536 different objects and 603 different object attributes. For binary question types this results in questions with solely affirmative answers. To produce an answer balanced dataset, we run a second stage of question argument generation for binary questions to generate plausible negative questions with false objects or attributes. We sample false objects from a distribution conditioned on an image's objects, and optionally sample object attributes from a distribution conditioned on the chosen object. For <choices> arguments, false choices are again generated from a distribution conditioned on the object's hypernym for Q5 questions, the attribute category for Q6 questions, or the action category for the Q7 questions. We additionally ensure that the generated choices are mutually exclusive (e.g. tan or beige would be an invalid gen-eration). To get more diverse multi-choice questions, we first generate a large pool of question candidates, and then select only a small number of questions sampled from this pool with sample probabilities inversely proportional to the count of the questions' hypernym, attribute class, or action class, and the count of the generated answer. Question argument refinement. To improve the reliability of generated questions, we apply a variety of checks and constraints. For example, when sampling false objects from the conditional distribution, we filter out all objects (and their hypernyms 3 ) present in the scene graph in order to guarantee that the sampled object is truly not present. We also filter out question arguments that are not included in the image scene graph but are sub-parts of objects that are annotated (e.g., tire when a car is annotated). Finally, we enforce various logical constraints on question arguments to prevent trivial or malformed questions. For example, for conjunctive and disjunctive questions (Q2, Q3), we apply a hypernym exclusion constraint to prevent questions like Is there a black cat and a black animal in the image?. We now describe our procedure for creating perturbed instances ( I 2 , q 2 , a 2 ) for the six tests. In all tests except visual obfuscation, the image remains unchanged, i.e. I 2 = I 1 . 3 We use an ontology with hypernym paths generated with WordNet (Miller, 1995). We manually review and revise the default synset annotations from Visual Genome (Krishna et al., 2017) for the entire object vocabulary, and compare to a sample of annotated images to ensure general validity. (a) Rephrasing invariance. Since each original question q 1 was generated using a text template, we simply use a different template of the same type to generate a valid rephrasing q 2 . The image and answer remain the same, i.e. I 1 = I 2 , a 1 = a 2 and the model is expected to be invariant to this rephrasing. We apply this to Q1, Q2, Q3, Q5, Q6 and Q7. (b) Ontological invariance. Here, we use object verification questions (Q1) only and perform two types of transformations. For positive questions (i.e. a 1 = yes ), we filter question arguments to only include objects that are valid hyponyms (using WordNet again) and use those to generate a perturbed question q 2 by changing the hyponym to a hypernym. For example, q 1 = Do you see a jogging woman ? with a 1 = yes is paired with q 2 = Do you see a jogging person ? containing a hypernym. Similarly, for negative questions ( a 1 = no ), we filter question arguments to only include valid hypernyms and generate a q 2 : thus for example, q 1 = Do you see a jogging person  with a 1 = no is paired with q 2 = Do you see a jogging woman ? containing a hyponym, a 2 = no also. (c) Order invariance. Order invariance tests apply to conjunctive verification, disjunctive verification, and all multi-choice question types; models are expected to be invariant to the logical order of arguments. We perturb conjunctive verification and disjunctive verification questions by swapping the questions' first and second arguments ( <obj1>, <obj2> ). For multi-choice question types, we perturb instances by generating the <choices> argument with different orders. The answer remains the same in both cases by construction. (d) Visual obfuscation invariance. For this test, we let q 1 = q 2 and a 1 = a 2 but generate a perturbed image I 2 by obscuring parts of I 1 that are irrelevant to the question at hand using bounding box annotations from Visual Genome (Krishna et al., 2017). For all true objects in a question, we consider the bounding boxes around these object(s) to be the foreground and all other pixels in the image to be the background . For negative verification questions asking about object(s) not present in the image, we select one (or two) random object bounding box(es) as the foreground and consider everything else to be the background, since focusing on any image region should not affect the model's answer. 4 We then apply five types of perturbations to obscure the background: (i-iii) Gaussian blurring using the soft masking method of (Yang et al., 2021) with light (  = 3 ), medium (  = 6 ), or heavy (  = 9 ) blurring, (iv) Masking with the channel-wise averaged pixel value from the GQA (Hudson and Manning, 2019) training dataset, entirely obscuring the context, and (v) Cropping to the smallest rectangular region including the foreground. Example images are shown in Appendix A.2. (e) Negation directional expectation. For the negation directional test, we use object verification, conjunctive verification, and disjunctive verification questions. Each question q 1 is perturbed by substituting the original's text template with a paired negated text template to create q 2 . Since each perturbed question represents the negation of the original, the expected answers a 1 (cid:54) = a 2 . (f) Attribute antonym directional expectation. We perturb the generated attribute verification questions by changing the <attr> question argument to its antonym using WordNet. All attribute antonym relations are manually curated to remove unintuitive examples; questions with arguments without a valid antonym are discarded. The original and perturbed questions of a pair have opposite answers a 1 (cid:54) = a 2 . To assess the quality, difficulty and validity of the generated tests, we sample 100 question pairs (200 questions) from each question type for the 6 tests and procure 5 annotations per question from workers on Amazon Mechanical Turk. Workers are vetted for a 97% approval rating and minimum of 500 completed HITs. Workers take  2 minutes per task on average and are compensated $0 . 50 per task and thus  $15 per hour. Each HIT include 24 questions total, including 4 verification questions 5 , and typically include a variety of question types from each of our tests. 4 We choose 32  32 as a minimum bounding box size, shown to be reliably recognized by humans (Torralba et al., 2008). 5 Tasks are also interspersed with binary or multi-choice gold-standard questions with perfect annotator agreement from the VQA dataset (Antol et al., 2015) which are required to be answered correctly before a HIT can be submitted. Workers are given the opportunity to correct answers before submitting if a gold-standard question has been answered incorrectly. Human agreement. In addition to yes and no for binary questions and the appropriate choices for multi-choice questions, all questions offer an ambiguous option. Human answers are the majority vote among the 5 workers; questions failing to reach majority or with ambiguous as the majority are always counted against accuracy. This process is inspired by the human evaluations of implied question pairs in Ribeiro et al. (2020a). We report both human and model performance in Section 6. 5.2 Evaluated models We evaluate six recent models on our tests, and compare them to human accuracy. Models are trained on the GQA (Hudson and Manning, 2019) balanced training split (using hyperparameters suggested from the original papers). All models, except LXMERT 6 , are trained and finetuned using the MMF (Singh et al., 2020) library and region of interest (RoI) features from Faster R-CNN (Ren et al., 2015) with a ResNeXt-152 (Xie et al., 2017) backbone pre-trained on the Visual Genome (Kr-ishna et al., 2017) dataset for object-based models. More details are provided in Appendix A.3. Model initialization and pre-training. Of the six models evaluated, a defining characteristic of each model relates to their initializations, image-encoding choice, and the use of multi-modal pretraining. Our most basic model (CNN+LSTM) is randomly initialized and uses no pre-trained components; however, GloVe (Pennington et al., 2014) word embeddings are used for representing tokens. Another class of models use pre-trained image encoders to extract object features from images. Of our models, BAN (Kim et al., 2018) is randomly initialized prior to training but ingests pre-trained Faster R-CNN features which should provide the model with enhanced visual capabilities over the CNN+LSTM model. MultiModal BiTransformer (MMBT) (Kiela et al., 2019), uses similar pre-trained image features as BAN but is further initialized with pre-trained BERT (Devlin et al., 2019) weights prior to training on GQA. The last class of models are multi-modal pretrained models; those that use pre-trained image features and are pre-trained on multi-modal tasks, such as image-based masked language modeling. Models in this class include LXMERT (Tan and 6 For LXMERT we use the authors' open source repository at https://github.com/airsplay/lxmert Bansal, 2020), ViLBERT (Lu et al., 2019), and VisualBERT (Li et al., 2019). Similar to MMBT, ViLBERT and VisualBERT are also initialized with pre-trained weights from BERT. Accuracy ( ACC ). On our test datasets with K paired instances, we define accuracy as: 1 2 KK (cid:88) i =1 1 [ a i 1 = a i 1 ] + 1 [ a i 2 = a i 2 ] where the model answers  a ia ,  a i 2 on the original and perturbed questions respectively are compared to the ground truth answers a i 1 , a i 2 . Self-consistency ( CONS ). We measure self-consistency of the model predictions across the original and perturbed questions as 1 KK (cid:88) i =1 (cid:26) 1 [ a i 1 =  a i 2 ] on invariance tests 1 [ a i 1 (cid:54) =  a i 2 ] on directional exp. tests Note that these metrics only measure the internal consistency of the model and do not include the ground truth answers a i 1 , a i 2 . Comprehensive accuracy ( C-ACC ). We define comprehensive accuracy as: 1 KK (cid:88) i =1 1 [ a i 1 = a i 1   a i 2 = a i 2 ] measuring whether model predictions are both accurate and self-consistent across perturbations. invariance and directionality tests. Fig 2 details the performance of various models under our suite of tests. Each bar in the figure shows both ACC and C-ACC for the model, with the arrow representing the gap between the two. We first observe that all models achieve significantly lower performance (at least a 8% drop) compared to humans (grey). Even simple tests such as REPHRASEINV (Figure 2(a)) prove to be quite challenging, with models managing < 68% ACC compared to humans' 86%. On tests like NEGATION-DIR (Fig-ure 2(e)), models only get about 50% accuracy, substantially worse than human scores of 78%. Moreover, C-ACC is considerably lower than ACC across the board, with as much as a 35% gap on NEGATION-DIR and 14.5% on ANTONYMDIR tests, even for a state-of-the-art model like LXMERT. 7 Even though this gap is smaller on 7 Other modern systems like ViLBERT and VisualBERT 6397 Figure 2: ACC and C-ACC across all six tests. other tests like REPHRASE-INV or VISUAL-INV , the performance drop is still at least 6-7% in most cases. This means that models are not invariant to textual rephrasings of questions and do not have a strong grasp of concepts like attributes and negation, despite negation of attributes appearing in the GQA training dataset. (R2) VQA systems are not self-consistent in their predictions. Table 1 shows the self-consistency scores for all models under our different tests. While humans achieve CONS > 88% in all the tests, VQA models are much worse  at least 6% lower CONS in all cases, with the best performing model (LXMERT) still 26% lower than human performance on average across tests and models. Scores are especially low on the directional tests (antonym and negation), which have even worse C-ACC . means that models are confused in their decisions simply with the addition of negation words  this hints at issues of overfitting to spurious feature without understanding the presence or absence of specific concepts, corroborating the findings of (Bitton et al., 2021). Interestingly, the best performing model (LXMERT) is not always the most consistent. Furthermore, there is no single model that is the most self-consistent, with LXMERT, ViLBERT and VisualBERT each returning the highest consistency scores on different tests. (R3) Models are more robust to hyponym than hypernym variations. Breaking out the results on the ontological invariance test (Figure 2 (c)) in the last two columns of Table 2, we see that self-consistency is higher on the hyponym perturbations (on negative answer questions) than on hypernym perturbations (positive questions); this 6398 Conjunction Disjunction ACCY N OACCY N O BAN 52 53 47 0 52 71 28 0 CNN+LSTM 39 53 47 0 35 65 35 0 LXMERT 78 49 51 0 56 59 32 9 MMBT 56 50 50 0 55 63 34 2 ViLBERT 58 54 46 0 56 79 21 0 VisualBERT 59 49 51 0 57 70 30 0 Table 3: Conjunctive (Con) vs disjunctive (Dis) comprehensive accuracy on ORDER-INV , along with a breakdown of response rates for yes (Y), no (N) and other than yes/no (O). effect is particularly noticeable for MMBT and ViLBERT with a 19% and 15% difference, respectively. Thus, when an object is not detected in an image its hyponym elicits a negative response as expected; however when an object (like steak) is detected, the hypernym question (Is there any meat in the image?) may trip the model to generate a negative response. This points to the need for more structured, hierarchical grounding of concepts in these models. (R4) Models perform better on conjunctive rather than disjunctive tests. From Table 3, we note that models generally have higher accuracy on conjunctive rather than disjunctive tests, with the largest discrepancy for LXMERT at 81% accuracy on conjunctive tests vs only 62% on disjunctive. Many models seem to exhibit a strong positive bias for disjunctive questions, suggesting they may just be short-cutting to answering yes' for disjunctive questions. LXMERT also seems to frequently confuse disjunctive questions for an open-ended or multi-choice question. provides a breakdown of LXMERT's scores for binary and multi-choice questions. It is evident that multi-choice questions are harder for the model, with self-consistency dropping by 16% between binary and multi-choice questions, and C-ACC dropping by 33%. This is surprising since the multi-choice questions only include two or three choices and hence are quite similar to the binary (yes/no) questions. This may indicate a bias in the models towards binary questions with simple answers. Furthermore, Table 4 also shows that models consistently perform worse on 3-choice questions than 2-choice ones, with even the top-performing LXMERT having a 7% drop from 62% to 55%. This hints that there may be some effect of randomness in the way these models pick their answers. In contrast and as expected, humans are robust to the number of choices. to deal with. From Figure 2 and Table 1, we notice that the models are slightly more robust to visual perturbations on average compared to the lexical ones. All models only have a drop of 4-8% from ACC to C-ACC , while self-consistency of all models is also 78% or higher. Appendix A.2 provides a more detailed breakdown of all the different visual perturbation tests we performed. (R7) Direct data augmentation improves CARETS evaluation. We show the feasibility of high performance on CARETS through data augmentation. We add 95,000 questions generated from CARETS question templates, and using a similar distribution of question types, to the original training split of GQA and re-train the LXMERT model. Table 5 shows that this dra-6399 matically improves the model on all three metrics ( ACC , self-consistency and C-ACC ), with the LXMERT(Augmented) model achieving near human performance on tests like ORDER-INV and ANTONYM-DIR . Since CARETS is designed to be an evaluation suite, these numbers show that CARETS questions should generally be within the capabilities of existing SOTA models, provided that they are able to generalize appropriately. In this work, we have developed CARETS  a new test suite for capability-focused robustness testing and comprehension evaluation of visual question answering (VQA) models. CARETS consists of six different tests that use instance pairs to evaluate models on their understanding of various linguistic and visual concepts. Using this test suite, we evaluated six modern VQA systems to reveal several inconsistencies in state-of-the-art models and provide a fine-grained view of their comprehension of different visuo-linguistic concepts. Quite surprisingly, we find that even state-of-the-art models struggle with concepts like negation, disjunction, order invariance and multi-choice questions. CARETS can also support the addition of more tests in the future and we view it as a platform for continually stress testing and improving VQA models. CARETS emulates previous work in using text templates to generate questions and their textual perturbations (Hudson and Manning, 2019; Johnson et al., 2017; Ribeiro et al., 2020b). The use of templates to generate perturbations is motivated by the desire to maintain the grounded integrity of generations, ensuring that they remain relevant and that the generated label is true in the context of the subject image. While we have sought to generate a diverse language set through using a large number of templates (nearly 200 in total), there are some limitations to this approach. An improvement to our approach may be able to generate more sophisticated questions and perturbations through conditional text generation (Schick and Schtze, 2021; Madaan et al., 2021; Wu et al., 2021) while also preserving our other motivations, such as atomicity and grounded relevancy. This material is based upon work supported by the National Science Foundation under Grant No. 2107048. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. We would also like to thank Tianyu Gao, Austin W. Hanjie and Alexander Wettig for their valuable feedback and advice.
method abstain method abstain method abstain abstain abstain other abstain abstain abstain method method method abstain result abstain abstain abstain objective other other other objective other method other method other objective other other other method method abstain abstain method abstain abstain abstain method other other abstain other method abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain method method abstain method method abstain method method abstain method abstain method abstain method method abstain abstain abstain abstain abstain method abstain abstain method method method method abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain other abstain abstain abstain method abstain method abstain abstain other abstain abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain objective abstain result result result abstain abstain method abstain other other other