We propose EASE , a simple diagnostic tool for Visual Question Answering (VQA) which quantifies the difficulty of an image, question sample. EASE is based on the pattern of answers provided by multiple annotators to a given question. In particular, it considers two aspects of the answers: (i) their Entropy; (ii) their Semantic content. First, we prove the validity of our diagnostic to identify samples that are easy/hard for state-of-art VQA models. Second, we show that EASE can be successfully used to select the most-informative samples for training/fine-tuning. Crucially, only information that is readily available in any VQA dataset is used to compute its scores. 1 1 Introduction Visual Question Answering (VQA; Antol et al., 2015) requires models to jointly understand an image and a natural language question. This is a challenging task; despite massive training data and recent pre-training strategies (Tan and Bansal, 2019; Lu et al., 2019; Chen et al., 2020) models still struggle to close the gap with oracle performance. VQA datasets (e.g., Goyal et al., 2017; Gurari et al., 2018) consist of (cid:104) image, question (cid:105) pairs for which N human annotators have provided an answer in natural language. When trained on these samples, VQA models are fed with the most frequently chosen answer in the pattern. During inference, the answer with the highest probability is evaluated against the pattern of N ground-truth answers. According to the standard VQA metric (An-tol et al., 2015), a model's prediction is considered as perfectly correct if it matches an answer that was frequent in the pattern; less accurate if matching an underrepresented one. This metric implies that, for the majority of cases, several annotators agree on the same exact answerand a model can thus achieve 100 % accuracy in the task. On the other 1 Code at: github.com/shailzajolly/EaSe Q: What is the pattern of the little girl's dress? hand, this suggests that various (cid:104) image, question (cid:105) pairs can have different patterns of answers; i.e., they can be more or less scattered depending on the features of the question, the image, or both. In Fig. 1, the annotators did not converge on the same answer for either of the two questions. However, while in the top question the 10 annotators provided semantically similar answers (e.g., plaid , plaid and floral , etc.), in the bottom one very different answers were given (e.g., road , sweden ). In line with recent work aimed at predicting the agreement between annotators (Gurari and Grau-man, 2017), the distribution of answers for a given (cid:104) image, question (cid:105) pair (Yang et al., 2018), or the difficulty of visual questions (Terao et al., 2020), in this paper we introduce EASE , a diagnostic tool for VQA which is based on the answers provided to a given question. We propose that two main features of the answer pattern, E ntropy a nd Se mantic content, are informative of the degree of difficulty of a sample. In particular, we conjecture that the more scattered an answer pattern, the more difficult the sample (Fig. 1, down)unless some or all of those answers are semantically similar (Fig. 1, top). By experimenting with various VQA datasets and models, we first assess the effectiveness of our diagnostic to identify the samples that are easy/difficult for a model. Second, we use EASE to select increasingly difficult subsets of data that we use to train/fine-tune our models, based on the hypothesis that difficult cases are also more informative during training. In both cases, we show that our simple method is very effective: (1) models are shown to struggle with the most difficult samples according to it; (2) training/fine-tuning models with only a little fraction of samplesthe most difficult onesmakes them achieve very high results, which are comparable to models trained/fine-tuned with the whole training data. Finally, EASE is shown to correlate with the confidence scores provided by human annotators along with their answers, which reveals that it captures a notion of difficulty in line with that by human speakers. We focus on (cid:104) image, question (cid:105) VQA samples and aim to quantify their difficulty , i.e., how challenging it is for a model to answer them correctly. We propose that the difficulty of a sample can be quanti-fied based on the (readily available) characteristics of the pattern of answers provided by the annotators, and devise a diagnostic tool that builds on this assumption. In particular, we focus on two aspects of the pattern: 1) its Entropy , i.e., how scattered it is in terms of the number of unique answer strings; 2) its Semantics , i.e., how (dis)similar are the answers in it with respect to their overall semantic representation. We name our diagnostic tool EASE and describe it in detail below. Entropy (E) We consider all the answers provided by the annotators for a given sample. Similar to Yang et al. (2018), we measure the Entropy of a pattern using Eq. 1: E ( p f ) =  1  M (cid:88) k =1 p k  log ( p k ) (1) where p f is the distribution of the M unique answers based on their frequency, and  is the highest possible Entropy value 2 that is used to normalize E in [0 , 1] . High E values (close to 1 ) are assigned to highly scattered distributions; vice versa , low values of E (close to 0 ) are assigned to highly con-2 In our data, the maximum Entropy value is equal to 2 . Semantics (SE ) E is based on the frequency of unique answer strings in a given pattern. As such, it treats various strings as different, regardless of whether strings are semantically similar. This, however, is crucial: answers to a given question that are semantically different reveal inconsistencies among annotators, which in turn is indicative of the difficulty of a sample. In contrast, semantically similar answers are a proxy for the ease of the sample, though these answers are different in their surface realization (see, e.g., a couple vs. a pair ). We use a simple method based on pre-trained word embeddings (Mikolov et al., 2018) to oper-ationalize SE . In particular, given a pattern of answers, we perform the following steps to reorganize it by aggregating semantically similar answers and their corresponding frequencies: (1) We compute a representation of each answer in the pattern by averaging its words embeddings, similar to Chao et al. (2018); (2) We build an answer's centroid, i.e., an average representation of all the unique answers that encodes the overall semantics in the pattern; (3) We compute the pairwise co-sine similarity ( cos ) between the centroid and each unique answer in the pattern (negative values are clamped to 0 to have similarity in [0 , 1] ); (4) We group together all the answers whose cos with the centroid embedding exceeds a certain threshold. The threshold  is dynamically set. It is computed at the datumlevel to adapt to the features of each datapoint, and is defined by:  = cos ( MAX , centroid )   (2) where  is a small positive number close to 0 (here we experiment with  = 0 . 0001 ), and MAX is the answer with the maximum frequency in the pattern. In case more than one MAX is present, the lowest  is used. Finally, we obtain a new distribution where the answers that are semantically consistent with the pattern's overall content (the centroid) are put together, and their frequencies are summed up. EASE diagnostic We take the new distribution of answers after applying SE , p se , and compute EASE , a single value in [0 , 1] which quantifies the ease of a VQA sample. We obtain it as follows: EASE ( p se ) = 1  E ( p se ) (3) Method Split VQA2.0 VizWiz T V T V EaSe TH 40522 19805 3201 522 (9%) (9%) (16%) (16%) BH 189281 92606 10443 1646 (43%) (43%) (52%) (52%) E 213954 101943 6356 1005 (48%) (48%) (32%) (32%) Entropy TH 108457 53230 11903 1897 (25%) (25%) (60%) (60%) BH 187287 90896 7337 1165 (42%) (42%) (36%) (37%) E 148013 70228 760 111 (33%) (33%) (4%) (3%) Total 443757 214354 20000 3173 (100%) (100%) (100%) (100%) Table 1: Top: Number of samples in the TH, BH, and E splits of VQA2.0 and VizWiz based on EaSe. We experiment with two models: BUTD (Ander-son et al., 2018) and LXMERT (Tan and Bansal, 2019) (LXM). BUTD uses a GRU to encode the input questions and to attend the image RoI features, enabling region-based attention to generate the answer. LXM is a transformer-based architecture pretrained on several language and vision tasks. We use it with the default hyper-parameters set in the original implementation. The models are trained (BUTD) or fine-tuned (LXM), and then evaluated, on the datasets described below. We experiment with VQA2.0 (Goyal et al., 2017) and VizWiz (VW; Gurari et al., 2018). We choose these two datasets since they are very different from each other, both in terms of the images (object-centered vs. everyday-life) and the type and purpose of the questions (written, crowdsourced vs. spoken, goal-oriented) they contain. This fundamental diversity is confirmed by a preliminary analysis 3 on the answers to the questions contained in 3 Further details in Appendix B. See also Jolly et al. (2018). In VQA2.0, 33 % of the questions are assigned the same answer string by all annotators; as for VizWiz, this percentage drops to only 3 %. We take this low agreement as a proxy for the difficulty of the samples in this (and any) dataset: the more disegreement, the harder. To preliminarly test our hypothesis, we compute the EASE value for each sample in the train/val partitions of the two datasets and assign the samples into 3 splits based on their EASE value (num-ber of samples per split in Tab. 1, top): (1) EASY (E) : EASE = 1.0; (2) BOTTOM-HARD (BH) : 0.5 < = EASE < 1.0; (3) TOP-HARD (TH) : EASE < 0.5. We then test our trained models on each of our validation splits. If our hypothesis is correct, models should struggle with the harder splits selected by our tool. Tab. 2 shows that all models BUTD, LXM and LXM-S, a version of LXM trained from scratch on the taskindeed achieve much lower performance on the hard splits; in TH, their accuracy is halved compared to the entire ( all ) data. Moreover, it is interesting to note that, for LXM, pretraining appears to be overall beneficial, with the pretrained version outperforming the non-pretrained one in both datasets and all splits, with a margin of around 8 points on the entire data. For comparison, we run the same analysis using Entropy (specifically, 1  Entropy ) instead of EASE . As can be seen in Table 1 (bottom), the two methods give rise to very different data dis-Model TD VQA2.0 VizWiz all TH BH E all TH BH EBUTD TH(R)* 50.14 20.46 53.34 53.0 42.75 24.91 40.57 55.58 BUTD TH 44.13 26.1 51.3 41.13 42.46 25.1 39.69 56.02 BUTD TH+BH 56.6 29.73 61.2 57.64 48.58 29.58 47.57 60.1 BUTD TH+BH+E 61.43 29.61 62.81 66.36 50.12 29.56 48.95 62.73 LXM TH(R)* 69.61 34.76 69.44 76.55 46.42 26.03 45.78 58.06 LXM TH 67.24 35.64 67.58 73.02 46.65 26.13 45.79 58.73 LXM TH+BH 69.85 37.05 70.63 75.52 51.65 30.29 50.07 65.36 LXM TH+BH+E 70.57 35.51 70.26 77.65 53.40 32.82 52.26 65.97 Table 3: Accuracy on each split of VQA2.0 and VizWiz obtained by gradually training models first on TH, then adding BH and finally adding E samples. tributions. For example, in the train partition of VQA2.0, Entropy assigns much more cases than EASE to the TH split (in proportion, 25% cases for Entropy vs. 9% for EASE ) and much less to the E one ( 33% Entropy vs. 48% EASE ). On the one hand, this confirms the crucial role of our semantic component in determining EASE scores. On the other hand, we notice that the results obtained by the three models on the splits defined by Entropy follow a less clear pattern compared to the EASE ones (see Tab. 4 in Appendix). For example, in VizWiz, both BUTD and LXM-S achieve higher results in BH compared to E, which indicates that Entropy is not as effective as our tool in measuring the difficulty of a sample. Finally, for sanity check, we also tested model performance on splits having the same size of EASE 's TH, BH and E but including random samples (see Tab. 5 in Appendix). The sampling was performed 10 times and results averaged. As expected, no difference in performance between the three splits was observed. Overall, this proof-of-concept analysis reveals that current SOTA modelsincluding the extensively pretrained LXMsuffer with samples that are deemed hard by EASE . This suggests that our diagnostic tool genuinely selects the most challenging samples of a dataset. An intuitive question is whether training a model with these hard samples can make models more robust. This is based on the intuition that challenging samples could be more informative during training compared to easy ones. We test this hypothesis in the next section, where we use the splits defined by EASE to train models in a HardFirst (HF) approach. In HF, we train our VQA models incrementally, first using TH samples only, then adding BH samples, and finally using all training samples. The weights for the first stage are initialized randomly; we load the model's weights from previous stages for each incremental stage. For VQA2.0, the percentage of samples for each stage is 9.13% (TH), 51.79% (TH+BH), and 100% ( ALL ), and for VizWiz is 16%, 68.22%, and 100%. We hypothesize that harder splits, i.e., with low EASE scores, contain richer multimodal information that could be more informative during a model's learning. For comparison, we also evaluate models in the TH(R) condition: we train/fine-tune models with a set of data (with the same size as TH) randomly sampled from the training set. We repeat the sampling 10 times, and report the average accuracy. Results in Tab. 3 support our hypotheses. (1) With only 52% of the training data (TH+BH), BUTD obtains 90% of all validation accuracy (VA) in VQA2.0 compared to the model trained on the Figure 2: Percentage of samples per question type in VQA2.0-train for each of the three splits used in the HF training regime. whole data (Table 2). This is even more pronounced in VizWiz, where using TH+BH during training ( 68% of total data) leads to a comparable performance as the one obtained with the whole training data. Similarly, LXM achieves 98% VA using only 52% of training data for VQA2.0, and 97% VA with 68% training data in VizWiz. (2) Compared to the TH(R) condition, models trained/fine-tuned with TH achieve higher results in the TH split of both VQA2.0 and VizWiz, which confirms that TH samples are particularly bene-ficial for dealing with challenging cases. At the same time, when evaluated on the entire data ( all ), they perform similarly to TH(R) in VizWiz and slightly worse than TH(R) in VQA2.0. This is to be expected: randomly sampling from VizWiz where 68% cases are either BH or THwill likely produce a more similar distribution to that of TH as compared to sampling from VQA2.0, where E cases are 48% of the total. Since proportions are the same in the validation set, training/fine-tuning with easier cases in VQA2.0 will have a positive impact on E, which will drive performance on all . Overall, these results indicate that the hard samples selected by EASE are more informative than easier ones and help models obtain comparable performance with significantly less training data. We explore whether the hard splits selected by EASE contain question types that are known to be particularly challenging for VQA models, e.g., count and whquestions. As can be seen in Fig. 2, a higher proportion of wh( Other ) and count ( Number ) questions is observed in the hardest split compared to the other splits of VQA2.0. 4 In contrast, polar questions ( Yes/No ) are poorly represented in TH, which indicates they are overall less challenging for humans and less informative for the models. We test whether EASE correlates with human intuition of when is difficult to answer a question. To this end, we use the confidence scores provided by annotators along with their answers in VQA2.0, 5 which self-evaluate whether annotators are confi-dent in providing their answer. We map confidence scores yes , maybe , no to 1 , 0 . 5 , and 0 , respectively, and compute the average confidence score for each sample. We then compute Spearman's correlation between confidence scores and EASE scores, and find a substantial positive correlation both in train (  = 0 . 49 ) and val (  = 0 . 48 ) sets. This trend is also clear in Fig. 3, where higher confidence scores correspond to increasingly higher EASE values. We present EASE , a simple diagnostic tool which quantifies the difficulty of a VQA sample based on its pattern of answers. We show that EASE selects the most informative samples of a dataset, which is helpful to train/fine-tune VQA models more efficiently with less, but highly-informative data. In future work, we plan to combine model prediction for difficulty estimation in EASE . Shailza Jolly was supported by the TU Kaiserslautern CS Ph.D. scholarship program, the BMBF project XAINES (Grant 01IW20005), and the NVIDIA AI Lab (NVAIL) program. Sandro was funded by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No. 819455 awarded to Raquel Fernndez).
objective abstain abstain abstain abstain objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other objective abstain objective method result abstain method method method abstain abstain method method method abstain method abstain abstain abstain abstain method method abstain method abstain method method method method abstain abstain method abstain method method abstain abstain method method method method abstain abstain abstain method other abstain abstain method method method method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method result method other other