Overcoming Language Priors in Visual Question Answering with Adversarial Regularization
Modern Visual Question Answering (VQA) models have been shown to rely heavily on superficial correlations between question and answer words learned during training -e.g. overwhelmingly reporting the type of room as kitchen or the sport being played as tennis, irrespective of the image. Most alarmingly, this shortcoming is often not well reflected during evaluation because the same strong priors exist in test distributions; however, a VQA system that fails to ground questions in image content would likely perform poorly in real-world settings.
In this work, we present a novel regularization scheme for VQA that reduces this effect. We introduce a question-only model that takes as input the question encoding from the VQA model and must leverage language biases in order to succeed. We then pose training as an adversarial game between the VQA model and this question-only adversary -discouraging the VQA model from capturing language biases in its question encoding. Further, we leverage this question-only model to estimate the increase in model confidence after considering the image, which we maximize explicitly to encourage visual grounding. Our approach is a model agnostic training procedure and simple to implement. We show empirically that it can improve performance significantly on a bias-sensitive split of the VQA dataset for multiple base models -achieving state-of-the-art on this task. Further, on standard VQA tasks, our approach shows significantly less drop in accuracy compared to existing bias-reducing VQA models.
Bias in Visual Question Answering dataset can harm the image-grounding of the model, by relaying on the question modality, and prone to failure in the general real-life case. The paper deals with this issue by discouraging the VQA model from capturing language biases in its question encoding, alternating the question embedding to reduce question-only model in an adversarial setup. Moreover, the paper deals with the nuance of language bias in VQA, by encouraging visual grounding via mutual information of image and answer given the question.     Strengths:  • Compared to GVQA method, the adversarial loss is simpler and more elegant.  • The mutual information between answer and image is interesting. To my knowledge this is original component. It captures the nuance in question bias for VQA, whether the bias is attributed to annotations, or it is a real-world bias, e.g, sky is blue. This component improves the results and also stabilize training over different hyper-parameters.  • Comprehensive evaluation on SOTA UpDn, and not only SAN. Also, evaluation on both VQA v2 and VQA v1.  • The paper is well organized and easy to read, the contributions clearly laid out.   Weaknesses:  • Evaluation on Oracle(Ours, SAN) reported only in the supplementary. In my opinion, it should be part of the paper. To me, this is the most significant score, significant model should be useful to improve the test score of VQA2.0 dataset, which is still far from being solved. If I understand correctly from the supplementary, your model improves by 3.46%, while Oracle(GVQA, SAN)  by 4.42%. The paper should discuss weaknesses as well.  • A diagnose of stronger f_Q networks will be interesting. A 2-NN network is way too naïve. Why not trying some sort of LSTM variation, which is the common practice? • I would appreciate qualitative analysis, i.e, showing images that your model succeeds on and GVQA failed.  • Technical information is missing, size of train/test split, training parameters: optimization method used, time of training, hardware, number of parameters for each model, framework used.  • I wonder, is it possible to run your model on the train+val/test split of VQA2.0, which usually not correlates with score on train/val split.  Conclusion: The paper discuss an interesting loss to deal with vqa bias while training.  While it works better on VQA-CP v2, surprisingly the Oracle version is weaker than GVQA, a better discussion is expected. Also technical information given is limited.  After author response:  I'm happy the authors agreed to add suggested important results to the revised version. I believe this paper will be a good contribution to the conference, and hopefully will enlighten a future research related to bias in QA tasks.