While sophisticated Visual Question Answering models have achieved remarkable success, they tend to answer questions only according to superficial correlations between question and answer. Several recent approaches have been developed to address this language priors problem. However, most of them predict the correct answer according to one best output without checking the authenticity of answers. Besides, they only explore the interaction between image and question, ignoring the semantics of candidate answers. In this paper, we propose a select-and-rerank (SAR) progressive framework based on Visual Entailment. Specifically, we first select the candidate answers relevant to the question or the image, then we rerank the candidate answers by a visual entailment task, which verifies whether the image semantically entails the synthetic statement of the question and each candidate answer. Experimental results show the effectiveness of our proposed framework, which establishes a new state-ofthe-art accuracy on VQA-CP v2 with a 7 . 55% improvement. 1 1 Introduction Visual Question Answering (VQA) task is a multimodal problem which requires the comprehensive understanding of both visual and textual information. Presented with an input image and a question, the VQA system tries to determine the correct answer in the large prediction space. Recently, some studies (Jabri et al., 2016; Agrawal et al., 2016; Zhang et al., 2016; Goyal et al., 2017) demonstrate that VQA systems suffer from the superficial correlation bias (i.e. language priors) caused by accidental correlations between answers and questions. As a result, traditional VQA models always output the  Corresponding author: Zheng Lin. most common answer(Selvaraju et al., 2019) of the input sample's question category, no matter what image is given. To address this language priors problem, various approaches have been developed. However, through exploring the characteristics of the existing methods, we find that whether the general VQA models such as UpDn(Anderson et al., 2018) and LXMERT(Tan and Bansal, 2019) or models carefully designed for language priors, as LMH(Clark et al., 2019) and SSL(Zhu et al., 2020), yield a non-negligible problem. Both models predict the correct answer according to one best output without checking the authenticity of answers. Besides, these models have not made good use of the semantics information of answers that could be helpful for alleviating the language-priors. As presented in Figure 1(a), quite a few correct answers usually occur at top N candidates rather than top one. Meanwhile, if the top N candidate answers are given, the image can further verify the visual presence/absence of concepts based on the combination of the question and the candidate answer. As shown in Figure 1(b), the question is about the color of the bat and two candidate answers are yellow and black. After checking the correctness of candidate answers, the wrong answer yellow which is contradicted with the image can be excluded and the correct answer black which is consistent with the image is confirmed. Nevertheless, this visual verification, which utilizes answer semantics to alleviate language priors, has not been fully investigated. In this paper, we propose a select-and-rerank (SAR) progressive framework based on Visual Entailment. The intuition behind the proposed framework comes from two observations. First, after excluding the answers unrelated to the question and image, the prediction space is shrunken and we can obtain a small number of candidate answers. Second, on the condition that a question and one of its candidate answer is bridged into a complete statement, the authenticity of this statement can be inferred by the content of the image. Therefore, after selecting several possible answers as candidates, we can utilize the visual entailment, consisting of image-text pairs, to verify whether the image semantically entails the synthetic statement. Based on the entailment degree, we can further rerank candidate answers and give the model another chance to find the right answer. To summarize, our contributions are as follows: 1. We propose a select-and-rerank progressive framework to tackle the language priors problem, and empirically investigate a range of design choices for each module of this framework. In addition, it is a generic framework, which can be easily combined with the existing VQA models and further boost their abilities. 2. We highlight the verification process between text and image, and formulate the VQA task as a visual entailment problem. This process makes full use of the interactive information of image, question and candidate answers. 3. Experimental results demonstrate that our framework establishes a new state-of-the-art accuracy of 66 . 73% , outperforming the existing methods by a large margin. Language-Priors Methods To address the language prior problem of VQA models, a lot of approaches have been proposed, which can be roughly categorized into two lines: (1) Designing Designing Specific Debiasing Models to Reduce Biases. Most works of this line are ensemble-based methods (Ramakrishnan et al., 2018; Grand and Belinkov, 2019; Belinkov et al., 2019; Cadene et al., 2019; Clark et al., 2019; Mahabadi and Henderson, 2019), among these, LMH(Clark et al., 2019) reduces all biases between question-answer pairs by penalizing the samples that can be answered without utilizing image content. (2) Data Augmentation to Reduce Biases. The main idea of such works (Zhang et al., 2016; Goyal et al., 2017; Agrawal et al., 2018) is to carefully construct more balanced datasets to overcome priors. For example, the recent method SSL(Zhu et al., 2020) first automatically generates a set of balanced question-image pairs, then introduces an auxiliary self-supervised task to use the balanced data. CSS(Chen et al., 2020a) balances the data by adding more complementary samples which are generated by masking objects in the image or some keywords in the question. Based on CSS, CL(Liang et al., 2020) forces the model to utilize the relationship between complementary samples and original samples. Unlike SSL and CSS which do not use any extra manual annotations, MUTANT(Gokhale et al., 2020) locates critical objects in the image and critical words in the question by utilizing the extra object-name labels, which directly helps the model to ground the textual concepts in the image. However, above methods only explore the interaction between the image and the question, ignoring the semantics of candidate answers. In this paper, we propose a progressive VQA framework SAR which achieves better interaction among the question, the image and the answer. Answer Re-ranking Although Answer Reranking is still in the infancy in VQA task, it has been widely studied for QA tasks like open-domain question answering, in which models need to answer questions based on a broad range of open-domains knowledge sources. Recent works (Wang et al., 2018b,a; Kratzwald et al., 2019) address this task in a two-stage manner: extract candidates from all passages, then focus on these candidate answers and rerank them to get a final answer. RankVQA(Qiao et al., 2020) introduces Answer Re-ranking method to VQA task. They design an auxiliary task which reranks candidate answers according to their matching degrees with the input image and off-line generated image captions. However, RankVQA still predicts the final answer from Figure 2: Overview of the progressive framework SAR. the huge prediction space rather than selected candidate answers. Figure 2 shows an overview of the proposed select-and-rerank (SAR) framework, which consists of a Candidate Answer Selecting module and an Answer Re-ranking module. In the Candidate Answer Selecting module, given an image and a question, we first use a current VQA model to get a candidate answer set consisting of top N answers. In this module, the answers irrelevant to the question can be filtered out. Next, we formulate the VQA as a VE task in the Answer Re-ranking module, where the image is premise and the synthetic dense caption(Johnson et al., 2016) (combination of the answer and the question ) is hypothesis. We use the cross-domain pre-trained model LXMERT(Tan and Bansal, 2019) as VE scorer to compute the entailment score of each image-caption pair, and thus the answer corresponding to the dense caption with the highest score is our final prediction. The Candidate Answer Selector (CAS) selects several answers from all possible answers as candidates and thus shrinks the huge prediction space. Given a VQA dataset D = { I i , Q i } Mi =1 with M samples, where I i  I , Q i  Q are the image and question of the i th sample and A is the whole prediction space consisting of thousands of answer categories. Essentially, the VQA model applied as CAS is a | A | -class classifier, and is a free choice in our framework. Given an image I i and a question Q i , CAS first gives the regression scores over all optional answers: P ( A | Q i , I i ) . Then CAS chooses N answers A  i with top N scores as candidates, which is concluded as follows: A  i = topN ( argsort ( P ( A | Q i , I i ))) (1) N (hyper-parameter) candidate answers A  i = [ A 1 i , A 2 i , ..., A Ni ] are selected for each ( I i , Q i ) pair by CAS, forming a dataset D (cid:48) = { I i , Q i , A ni } M ,N i =1 ,n =1 with M  N instances, where A ni  A  i , for the next Answer Re-ranking module. In this paper, we mainly use SSL as our CAS. We also conduct experiments to analyze the impact of different CAS and different N . Visual Entailment (VE) task is proposed by Xie et al. (2019), where the premise is a real-world image, denoted by P image , and the hypothesis is a text, denoted by H text . Given a sample of ( P image , H text ), the goal of VE task is to determine whether the H text can be concluded based on the information of P image . According to following protocols, the label of the sample is assigned to (1) Entailment , if there is enough evidence in P image to conclude H text is true. (2) Contradiction , if there is enough evidence in P image to conclude H text is false. (3) Neutral , if there is no sufficient evidence in P image to give a conclusion about H text . A question Q i and each of its candidate answers A  i can be bridged into a complete statement, and then the image could verify the authenticity of each statement. More specifically, the visual presence of concepts (e.g. black bat/yellow bat) based on the combination of the question and the correct/wrong candidate answer can be en-tailed/contradicted by the content of the image. In this way, we achieve better interaction among question, image and answer. Therefore, we formulate VQA as a VE problem, in which the image I i is premise, and the synthetic statement of an answer A ni in A  i and question Q i , represented as ( Q i , A ni ), is hypothesis. For an image, synthetic statements of different questions describe different regions of the same image. Following Johnson et al. (2016), we also refer to the synthetic statement as dense caption. We use A + i to represent the A ni if A ni is the correct answer of Q i , use A  i otherwise. There is enough evidence in I i to prove ( Q i , A + i ) is true, i.e. the visual linguistic semantically entails ( Q i , A + i ). And there is enough evidence in I i to prove ( Q i , A  i ) is false, i.e. the visual linguistic semantically contradicts ( Q i , A  i ). Note that, there is no Neutral label in our VE task and we only have two labels: Entailment and Contradiction . We re-rank dense captions by contrastive learning, that is, ( Q i , A + i ) should be more semantically similar to I i than ( Q i , A  i ). The right part of Figure 2 illustrates this idea. The more semantically similar I i to ( Q i , A ni ), the deeper the visual entailment degree is. We score the visual entailment degree of I i to each ( Q i , A ni )  ( Q i , A  i ) and rerank the candidate answers A  i by this score. The ranking-first answer is our final output. Question-Answer Combination Strategy The answer information makes sense only when combine it with the question. We encode the combination of question and answer text to obtain the joint concept. We design three question-answer combination strategies: R , C and R  C to combine question and answer into synthetic dense caption C i : R : Replace question category prefix with answer . The prefix of each question is the question category such as are there, what color, etc. For instance, given a question How many flowers in the vase?, its answer 8 and its question category how many, the resulting dense caption is 8 flowers in the vase. Similarly, No a crosswalk is the result of question  Is this a crosswalk? and answer No. We build a dictionary of all question categories of the train set, then we adopt a Forward Maximum Matching algorithm to determine the question category for every test sample. C : Concatenate question and answer directly. For two cases above, the resulting dense captions are 8 How many flowers in the vase? and No Is this a crosswalk?. The resulting dense captions after concatenation are actually rhetorical questions. We deliberately add answer text to the front of question text in order to avoid the answer being deleted when trimming dense captions to the same length. R  C : We first use strategy R at training, which is aimed at preventing the model from excessively focusing on the co-occurrence relation between question category and answer, and then use strategy C at testing to introduce more information for inference. Adopting any strategy above, we combine Q i and each answer in A  i to derive the dense captions C  i . And thus we have a dataset D (cid:48)(cid:48) = { Ii, C ni } M ,N i =1 ,n =1 with M  N instances for VE task. VE Scorer We use the pre-trained model LXMERT to score the visual entailment degree of ( I i , C ni ). LXMERT separately encodes image and caption text in two streams. Next, the separate streams interact through co-attentional transformer layers. In the textual stream, the dense caption is encoded into a high-level concept. Then the visual representations from visual stream can verify the visual presence/absence of the high-level concept. We represent the VE score for the i th image and its n th candidate caption as: sigmoid ( T rm ( I i , C ni )) , where T rm () is the 1-demensional output from the dense layers following LXMERT,  () denotes the sigmoid function. The larger score represents higher entailment degree. We optimize parameters by minimizing the multi-label soft loss: LV E =  1 M  NM (cid:88) i =1 N (cid:88) n =1 [ t ni log (  ( T rm ( I i , C ni ))) + (1  t ni ) log (1   ( T rm ( I i , C ni )))] (2) where t ni is the soft target score of the n th answer. Combination with Language-Priors Method After Candidate Answer Selecting, the amount of candidate answers decreases from all possible answers to top N . Although some unrelated answers are filtered out, the dataset D (cid:48)(cid:48) for VE system is still biased. Therefore, we can optionally apply existing language-priors methods to our framework for further reducing language priors. Take the SSL as an example, we apply the loss function of its self-supervised task to our framework by adjusting the loss function to: L ssl =  M  NM (cid:88) i =1 N (cid:88) n =1 P ( I (cid:48) i , C ni ) (3) where ( I (cid:48) i , C ni ) denotes the irrelevant image-caption pairs,  is a down-weighting coefficients. Question Type Discriminator Intuitively, most Yes/No questions can be answered by the answer Yes or No. There is no need to provide too many candidate answers for Yes/No questions at the test stage. Therefore, we propose a Question Type Discriminator(QTD) to determine the question type and then correspondingly set different numbers of candidate answers, denoted as N (cid:48) . Specifically, we roughly divided question types (in-cluding Yes/No, Num and Other) into yes/no and non-yes/no. A GRU binary classifier is trained with cross-entropy loss and evaluated with 5-fold cross-validation on the train split of each dataset. Then, the trained QTD model with an accuracy about 97% is implemented as an off-line module during the test stage. We will further investigate the effect of N (cid:48) on each question type in the next section. The answer  A i corresponding to  C i is the final prediction. Datasets Our models are trained and evaluated on the VQA-CP v2(Agrawal et al., 2018) dataset, which is well-crafted by re-organizing VQA v2(Goyal et al., 2017) training and validation sets such that answers for each question category (65 categories according to the question prefix) have different distributions in the train and test sets. Therefore, VQA-CP v2 is a natural choice for evaluating VQA model's generalizability. The questions of VQA-CP v2 include 3 types: Yes/No, Num and Other. Note that the question type and question category (e.g.what color) are different. Besides, we also evaluate our models on the VQA v2 validation set for completeness, and compare the accuracy difference between two datasets with the standard VQA evaluation metric(Antol et al., 2015). Baselines We compare our method with the following baseline methods: UpDn(Anderson et al., 2018), AReg(Ramakrishnan et al., 2018), RUBi(Cadene et al., 2019), LMH(Clark et al., 2019), RankVQA(Qiao et al., 2020), SSL(Zhu et al., 2020), CSS(Chen et al., 2020a), CL(Liang et al., 2020) and LXMERT(Tan and Bansal, 2019). Most of them are designed for the language priors problem, while LXMERT represents the recent trend towards utilizing BERT-like pre-trained mod-els(Li et al., 2019; Chen et al., 2020b; Li et al., 2020) which have top performances on various downstream vision and language tasks (including VQA-v2). Note that MUTANT(Gokhale et al., 2020) uses the extra object-name label to ground the textual concepts in the image. For fair comparison, we do not compare with MUTANT. In this paper, we mainly choose SSL as our CAS and set N =12 and N =20 for training. To extract image features, we follow previous work and use the pre-trained Faster R-CNN to encode each image as a set of fixed 36 objects with 2048-dimensional feature vectors. We use the tokenizer of LXMERT to segment each dense caption into words. All the questions are trimmed to the same length of 15 or 18, respectively for R or C question-answer combination strategy. In the Answer Re-ranking Module, we respectively incorporate two language-priors methods, SSL and LMH, into our proposed framework SAR, which is dubbed as SAR+SSL and SAR+LMH. Our models are trained on two TITAN RTX 24GB GPUs. We train SAR+SSL for 20 epochs with batch size of 32, SAR and SAR+LMH for 10 epochs with batch size of 64. For SAR+SSL, we follow the same setting as the original paper(Zhu et al., 2020), except that we don't need to pre-train the model with the VQA loss before fine-tuning it with the self-supervised loss. The Adam optimizer is adopted with the learning rate 1e5. For Question Type Discriminator, we use 300-dimensional Glove(Pennington et al., 2014) vectors to initialize word embeddings and feed them into a unidirectional GRU with 128 hidden units. When testing on the VAQ-CP v2, N (cid:48) ranges from 1-2 for yes/no questions and 5-15 for non-yes/no questions. As for VQA v2, N (cid:48) ranges from 1-2 for yes/no Model VQA-CP v2 test(%)  VQA-v2 val(%)  GAP ALL Yes / No Num Other All Yes / No Num Other (%)  UpDN(Anderson et al., 2018) 39.74 42.27 11.93 46.05 63.48 81.18 42.14 55.66 23.74 Areg(Ramakrishnan et al., 2018) 41.17 65.49 15.48 35.48 62.75 79.84 42.35 55.16 21.58 RUBI(Cadene et al., 2019) 47.11 68.65 20.28 43.18 61.16 --14.05 LMH(Clark et al., 2019) 52.45 69.81 44.46 45.54 61.64 77.85 40.03 55.04 9.19 RankVQA(Qiao et al., 2020) 43.05 42.53 13.91 51.32 65.42 82.51 57.75 45.35 22.37 LXMERT(Tan and Bansal, 2019) 46.23 42.84 18.91 55.51 74.16 89.31 56.85 65.14 27.93 SSL(Zhu et al., 2020) 57.59 86.53 29.87 50.03 63.73 --6.14 CSS(Chen et al., 2020a) 58.95 84.37 49.42 48.21 59.91 73.25 39.77 55.11 0.96 CL(Liang et al., 2020) 59.18 86.99 49.89 47.16 ---Top12-SAR(R  C) ( Ours ) 64.55 83.03 50.05 58.8 70.41 87.87 54.34 61.38 5.86 Top20-SAR(R  C) ( Ours ) 65.44 83.13 54.52 59.16 70.63 87.91 54.93 61.64 5.19 Top12-SAR+SSL(R  C) ( Ours ) 64.29 82.86 51.98 57.94 69.84 87.22 54.41 60.70 5.55 Top20-SAR+SSL(R  C) ( Ours ) 65.32 83.41 54.32 58.85 70.03 87.47 54.59 60.85 4.71 Top12-SAR+LMH(R) ( Ours ) 65.93 85.38 62.30 56.73 69.13 87.61 50.43 60.03 3.20 Top20-SAR+LMH(R) ( Ours ) 66.73 86.00 62.34 57.84 69.22 87.46 51.20 60.12 2.49 Table 1: Results on VQA-CP v2 test and VQA-v2 validation set. questions and 2-5 for non-yes/no questions. Performance on two benchmarks VQA-CP-v2 and VQA-v2 is shown in Table 1. We report the best results of SAR, SAR+SSL and SAR+LMH among 3 question-answer combination strategies respectively. TopNrepresents that N candidate answers (selected by CAS) feed into the Answer Reranking Module for training. Our approach is evaluated with two settings of N (12 and 20). From the results on VQA-CP v2 shown in Table 1, we can observe that: (1) Top20-SAR+LMH establishes a new state-of-the-art accuracy of 66 . 73% on VQA-CP v2, beating the previous best-performing method CL by 7 . 55% . Even without combining language-priors methods in Answer Re-ranking module, our model Top20-SAR outperforms CL by 6 . 26% . These show the outstanding effectiveness of our proposed SAR framework. (2) SAR+SSL and SAR+LMH achieve much better performance than SSL and LMH, which demonstrates that SAR is compatible with current language-priors methods and could realize their full potential. (3) Compared with another reranking-based model RankVQA, our method elevates the performance by a large margin of 23 . 68% . This shows the superiority of our proposed progressive select-and-rerank framework over RankVQA which only uses the answer reranking as an auxiliary task. (4) Previous models did not generalize well on all question types. CL is the previous best on the Yes/No, Num questions and LXMERT on the Other questions. In comparison, our model not only rivals the previous best model on the Yes/No questions but also improves the best performance on the Num and Other questions by 12 . 45% and 3 . 65% . The remarkable performance on all question types demonstrates that our model makes a significant progress toward a truly comprehensive VQA model. We also evaluate our method on the VQA v2 which is deemed to have strong language biases. As shown in Table 1, our method achieves the best accuracy of 70 . 63% amongst baselines specially designed for overcoming language priors, and is the closest to the SOTA established by LXMERT which is trained explicitly for the biased data setting. For completeness, the performance gap between two datasets is also compared in Table 1 with the protocol from Chen et al. (2020a). Compared with most previous models which suffer severe performance drops between VQA v2 and VQA-CP v2 (e.g., 27 . 93% in LXMERT), the Top20-SAR+LMH significantly decreases the performance drop to 2 . 49% , which demonstrates the effectiveness of our framework to further overcome the language biases. Though CSS achieves a better performance gap, it sacrifices the performance on the VQA v2. Meanwhile, as N rises from 12 to 20, our models achieve better accuracy on both datasets along with a smaller performance gap. This demonstrates that, unlike previous methods, our method can alleviate language priors while maintaining an excellent capability of answering questions. Nonetheless, we Figure 3: Results from model SAR+SSL(R  C) in VQA-CP v2 with different N during training. believe that, how to improve the model's generality and further transform the trade-off between eliminating language priors and answering questions into winwin outcomes, is a promising research direction in the future. From Figure 3, we can observe that the overall performance is getting better as N increases. The performance improvement on the Num and Other questions is especially obvious, and there is a very slight drop on the Yes/No questions. We believe that SAR can further get better performance by properly increasing N . Due to the resource limitation, the largest N we use is 20 in this paper. To find out the potential performance limitation of CAS models, we show the accuracy of 3 CAS models on the VQA-CP v2 test set. As shown in Figure 1 (a), the Top3 accuracy (acc) of 3 models is about 70% and Top6 acc is 80% , which guarantees that sufficient correct answers are recalled by CAS. And thus, the performance limitation of CAS is negligible. We also conduct experiments to investigate the effect of different CAS on SAR. From the results shown in Table 2, we can observe that: (1) Choosing a better VQA model as CAS does not guarantee a better performance, e.g. performance based on Top N Model R C R  C Top12 SAR 59.51 60.24 64.55 SAR+SSL 62.12 62.87 64.29 SAR+LMH 65.93 65.23 65.14 Top20 SAR 60.43 61.86 65.44 SAR+SSL 62.29 63.94 65.32 SAR+LMH 66.73 65.19 66.71 Table 3: Results on the VQA-CP v2 test set based on different question-answer combination strategies: R, C and R  C. The major difference between R and C is whether keeping question prefix which includes 65 categories. UpDn outperforms that based on LMH, but LMH is a better VQA model in overcoming language priors compared with UpDn. This is because a good Candidate Answer Selector has two requirements: (a) It should be able to recall more correct answers. (b) Under the scenario of language biases, wrong answers recalled by CAS at training time should have superficial correlations with the question as strong as possible. However, the ensemble methods, such as LMH, are trained to pay more attention to the samples which are not correctly answered by the question-only model. This seriously reduces the recall rate of those language-priors wrong answers, which leads to the training data for VE is too simple and thus hurts the model's capability of reducing language priors. (2) If CAS is the general VQA model UpDn rather than LMH and SSL, the improvement brought from the combination with language-priors method in Answer Re-ranking module is more obvious. (3) Even we choose the UpDn, a backbone model of most current works, as our CAS and do not involve any language-priors methods , SAR still achieves a much better accuracy than the previous SOTA model CL by 2 . 53% , which shows that our basic framework already possesses outstanding capability of reducing language priors. From the results shown in Table 3, we can observe that: (1) From overall results, R  C achieves or rivals the best performance on three models. On average, R  C outperforms C by 2 . 02% which demonstrates avoiding the co-occurrence of question category and answer during training time could effectively alleviate language priors; R  C outperforms R by 2 . 41% which indicates that the informa-Model All Yes / No Num Other LXM 46.23 42.84 18.91 55.51 LXM+SSL 53.09 55.07 29.60 58.50 CAS+LXM(R) 55.58 70.91 29.14 54.81 CAS+LXM+SSL(R) 59.41 76.60 40.81 55.51 CAS+LXM+QTD(R) 59.51 83.20 29.17 55.42 CAS+LXM+SSL+QTD(R) 62.12 85.14 41.63 55.68 Table 4: Ablation study to investigate the effect of each component of Top12-SAR+SSL: Candidate Answer Selector (CAS), LXMERT (LXM), Question Type Discriminator (QTD) and SSL. tion of question category is useful in inference. (2) On the SAR and SAR+SSL, C consistently outperforms R, but on the SAR+LMH, we see opposite results. This is probably because our method and the balancing-data method SSL could learn the positive bias resulted from the superficial correlations between question category and answer, which is useful for generalization, but the ensemble-based method LMH will attenuate positive bias during de-biasing process. (3) Even without language priors method, SAR with R  C rivals or outperforms the SAR+SSL and SAR+LMH with R or C, which shows that R  C strategy could help the model to alleviate language priors. As a result, compared with R or C, our framework with R  C only gains a slight performance improvement after using the same language-priors methods. CAS+ represents we use the select-and-rerank framework. From Table 4, we can find that: (1) LXM+SSL represents directly applying SSL to LXMERT. Its poor performance shows that the major contribution of our framework does not come from the combination of the language-priors method SSL and pre-trained model LXMERT. (2) Compared with LXM and LXM+SSL, CAS+LXM and CAS+LXM+SSL respectively gain prominent performance boost of 9 . 35% and 6 . 32% , which demonstrates the importance and effectiveness of our proposed select-and-rerank procedure. (3) CAS+LXM+QTD(R) and CAS+LXM+SSL+QTD(R) respectively outperform CAS+LXM(R) and CAS+LXM+SSL(R) by 3 . 93% and 2 . 71% , which shows the contribution of QTD module. This further demonstrates that choosing appropriate N (cid:48) for different question types is a useful step for model performance. (4) CAS+LXM+SSL+QTD improves the performance of CAS+LXM+QTD by 2 . 61% , which shows that Figure 4: Results from SAR(R), SAR+SSL(R), SAR(R  C) and SAR+LMH(R) with different N (cid:48) during test. From Figure 4, we can find that: (1) The best N (cid:48) for yes/no questions is smaller than that for non-yes/no questions due to the nature of yes/no question. (2) As N (cid:48) increases, the accuracy of Num and Other questions rises first and then decreases. There is a trade-off behind this phenomenon: when N (cid:48) is too small, the correct answer may not be recalled by CAS; when N (cid:48) is too large, the distraction from wrong answers makes it more difficult for model to choose the correct answer. We qualitatively evaluate the effectiveness of our framework. As shown in Figure 5, compared with SSL, SAR performs better not only in question answering but also in visual grounding. With the help of answer semantics, SAR can focus on the region relevant to the candidate answer and further use the region to verify its correctness. In this paper, we propose a select-and-rerank (SAR) progressive framework based on Visual Entailment. Specifically, we first select candidate answers to shrink the prediction space, then we rerank candidate answers by a visual entailment task which verifies whether the image semantically entails the synthetic statement of the question and each candidate answer. Our framework can make full use of the interactive information of image, question and candidate answers. In addition, it is a generic framework, which can be easily combined with the existing VQA models and further boost their abilities. We demonstrate advantages of our framework on the VQA-CP v2 dataset with extensive experiments and analyses. Our method establishes a new state-of-the-art accuracy of 66 . 73% with an improvement of 7 . 55% on the previous best. This work was supported by National Natural Science Foundation of China (No. 61976207, No. 61906187)
abstain abstain abstain abstain objective objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain objective abstain result result objective abstain result abstain objective abstain other other other other other other other other other other objective other other other other other other abstain method abstain method method abstain abstain method abstain other method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method abstain method abstain method abstain abstain abstain abstain method abstain method abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain result abstain result abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result result result abstain abstain other abstain result result abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain objective objective abstain abstain objective objective abstain abstain other