summaries: This paper proposes a new loss terms to encourage Transformer-based models perform "explicit reasoning", which is beneficial for transferring knowledge from oracle-trained VQA models to deployable ones. Specifically, the novel loss is similar with the one commonly used in modular network (e.g. converting questions to executable program/functions), but instead of relying on the predicted programs to assemble executable programs during inference, in this paper this loss is only used during training as an auxiliary regularization on top of regular Transformer-based VQA models. The authors provide theoretical proof about the benefits of using the additional loss, and empirically evaluated the advantage on GQA dataset (including out-of-distribution evaluation on GQA-OOD). The proposed system demonstrates not the best but still competitive performance especially considering the model size and training efficiency (without using extra data).  ========================================================================================  Thanks for the response from the authors. I've updated my rating after reading the rebuttal as well as the comments from other reviewers. The paper studies the problem of training VQA models on noisy/imperfect visual input (e.g., representations from existing object detectors). It proposes using program supervision as an additional task to assist in transferring reasoning patterns learned through clean/oracle visual inputs to the imperfect inputs. The experiments are performed on the GQA/GQA-OOD benchmarks, and the empirical results show gains on both frequent (head) and infrequent (tail) classes, showing improvement in reasoning skills. Furthermore, the work theoretically shows that predicting programs can reduce sample complexity. This paper targets on the application problem of visual question answering, whose recent state of the art was set by attention (typically transformer) - based deep neural networks. This work propose to use program supervision to introduce extra loss terms in the training of these transformer based models, arguing that the guidance from these signals improve the model's robustness against noise and fluctuations in the visual input during transfer learning. Theoretical analysis and experimental evaluations are conducted on program supervision, leading to plausible findings and results. This paper describes a novel approach for doing more robust VQA that doesn’t just learn to exploit the biases in the data. They build on previous work which demonstrated that training a model on oracle visual data makes it less susceptible to learning biases in the data. While the previous work showed that the better reasoning skills learnt by the models trained on oracle visuals decline when transferred to the noisier visuals extracted by pre-trained vision models, here the authors demonstrate that adding the additional supervision step of making the model predict the reasoning program alongside the VQA answer makes it more robust to such distributional shifts. 
abstractText: Methods for Visual Question Anwering (VQA) are notorious for leveraging dataset biases rather than performing reasoning, hindering generalization. It has been recently shown that better reasoning patterns emerge in attention layers of a stateof-the-art VQA model when they are trained on perfect (oracle) visual inputs. This provides evidence that deep neural networks can learn to reason when training conditions are favorable enough. However, transferring this learned knowledge to deployable models is a challenge, as much of it is lost during the transfer. We propose a method for knowledge transfer based on a regularization term in our loss function, supervising the sequence of required reasoning operations. We provide a theoretical analysis based on PAC-learning, showing that such program prediction can lead to decreased sample complexity under mild hypotheses. We also demonstrate the effectiveness of this approach experimentally on the GQA dataset and show its complementarity to BERT-like self-supervised pre-training.
authors.name: Corentin Kervadec
authors.name: Christian Wolf
authors.name: Grigory Antipov
authors.name: Moez Baccouche
authors.name: Madiha Nadri
references.authors: Aishwarya Agrawal Dhruv Batra Devi Parikh Aniruddha Kembhavi
references.title: Don’t just assume; look and answer: Overcoming priors for visual question answering
references.authors: Peter Anderson Xiaodong He Chris Buehler Damien Teney Mark Johnson Stephen Gould Lei Zhang
references.title: Bottom-up and top-down attention for image captioning and visual question answering
references.venue: In CVPR,
references.authors: Jacob Andreas Marcus Rohrbach Trevor Darrell Dan Klein
references.title: Neural module networks
references.venue: In CVPR,
references.authors: Stanislaw Antol Aishwarya Agrawal Jiasen Lu Margaret Mitchell Dhruv Batra C Lawrence Zitnick Devi Parikh
references.title: Vqa: Visual question answering
references.venue: In ICCV,
references.authors: Hedi Ben-Younes Rémi Cadene Matthieu Cord Nicolas Thome
references.title: Mutan: Multimodal tucker fusion for visual question answering
references.venue: In ICCV,
references.authors: Wenhu Chen Zhe Gan Linjie Li Yu Cheng William Wang Jingjing Liu
references.title: Meta module network for compositional visual reasoning
references.venue: In WACV,
references.authors: Kyunghyun Cho Bart van Merrienboer Çaglar Gülçehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio
references.title: Learning phrase representations using rnn encoder-decoder for statistical machine translation
references.venue: In EMNLP,
references.authors: Corentin Dancette Remi Cadene Damien Teney Matthieu Cord
references.title: Beyond question-based biases: Assessing multimodal shortcut learning in visual question answering
references.venue: arXiv preprint arXiv:2104.03149,
references.authors: Abhishek Das Harsh Agrawal C. Lawrence Zitnick Devi Parikh Dhruv Batra
references.title: Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions
references.authors: Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova
references.title: Bert: Pre-training of deep bidirectional transformers for language understanding
references.venue: In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,
references.authors: Robert Geirhos Jörn-Henrik Jacobsen Claudio Michaelis Richard Zemel Wieland Brendel Matthias Bethge Felix A Wichmann
references.title: Shortcut learning in deep neural networks
references.venue: Nature Machine Intelligence,
references.authors: Yash Goyal Tejas Khot Douglas Summers-Stay Dhruv Batra Devi Parikh
references.title: Making the v in vqa matter: Elevating the role of image understanding in visual question answering
references.venue: In CVPR,
references.authors: Ronghang Hu Jacob Andreas Marcus Rohrbach Trevor Darrell Kate Saenko
references.title: Learning to reason: End-to-end module networks for visual question answering
references.venue: In Proceedings of the IEEE International Conference on Computer Vision,
references.authors: Drew Hudson Christopher D Manning
references.title: Learning by abstraction: The neural state machine
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Drew A Hudson Christopher D Manning
references.title: Gqa: A new dataset for real-world visual reasoning and compositional question answering
references.venue: In CVPR,
references.authors: Justin Johnson Bharath Hariharan Laurens van der Maaten Li Fei-Fei C Lawrence Zitnick Ross Girshick
references.title: Clevr: A diagnostic dataset for compositional language and elementary visual reasoning
references.venue: In CVPR,
references.authors: Justin Johnson Bharath Hariharan Laurens Van Der Maaten Judy Hoffman Li Fei-Fei C Lawrence Zitnick Ross Girshick
references.title: Inferring and executing programs for visual reasoning
references.venue: In Proceedings of the IEEE International Conference on Computer Vision,
references.authors: C. Kervadec G. Antipov M. Baccouche C. Wolf
references.title: Roses Are Red, Violets Are Blue.
references.venue: Should VQA Expect Them To? Pre-print: arxiv:2006.05121,
references.authors: Corentin Kervadec Grigory Antipov Moez Baccouche Christian Wolf
references.title: Weak supervision helps emergence of word-object alignment and improves vision-language tasks
references.venue: In European Conference on Artificial Intelligence,
references.authors: Corentin Kervadec Grigory Antipov Moez Baccouche Christian Wolf
references.title: Roses are red, violets are blue... but should vqa expect them to
references.authors: Corentin Kervadec Theo Jaunet Grigory Antipov Moez Baccouche Romain Vuillemot Christian Wolf
references.title: How transferable are reasoning patterns in vqa
references.authors: Jin-Hwa Kim Jaehyun Jun Byoung-Tak Zhang
references.title: Bilinear attention networks
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Jiayuan Mao Chuang Gan Pushmeet Kohli Joshua B. Tenenbaum Jiajun Wu
references.title: The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision
references.venue: In International Conference on Learning Representations,
references.authors: Hubert Ramsauer Bernhard Schäfl Johannes Lehner Philipp Seidl Michael Widrich Lukas Gruber Markus Holzleitner Milena Pavlović Geir Kjetil Sandve Victor Greiff
references.title: Hopfield networks is all you need
references.authors: Shaoqing Ren Kaiming He Ross Girshick Jian Sun
references.title: Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing
references.authors: S.S. Du S. Arora W. Hu Z. Li R. Wang
references.title: Fine-grained Analysis of optimization and generalization for overparametrized two-layer neural networks
references.authors: Shai S. Shalev-Shwart S. Ben-David
references.title: Understanding Machine Learning - From Theory to Algorithms
references.authors: Robik Shrestha Kushal Kafle Christopher Kanan
references.title: A negative case analysis of visual grounding methods for vqa
references.venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
references.authors: Robik Shrestha Kushal Kafle Christopher Kanan
references.title: An investigation of critical issues in bias mitigation techniques
references.venue: arXiv preprint arXiv:2104.00170,
references.authors: Hao Tan Mohit Bansal
references.title: Lxmert: Learning cross-modality encoder representations from transformers
references.venue: In EMNLP,
references.authors: Damien Teney Ehsan Abbasnejad Kushal Kafle Robik Shrestha Christopher Kanan Anton van den Hengel
references.title: On the value of out-of-distribution testing: An example of goodhart's law
references.venue: NeurIPS,
references.authors: L.G. Valiant
references.title: A theory of the learnable
references.venue: In Communications of the ACM,
references.authors: Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Łukasz Kaiser Illia Polosukhin
references.title: Attention is all you need
references.venue: In Advances in neural information processing systems,
references.authors: K. Xu J. Li M. Zhang S.S. Du K.-I. K S. Jegelka
references.title: What can Neural Networks Reason About
references.venue: In ICLR,
references.authors: Kexin Yi Jiajun Wu Chuang Gan Antonio Torralba Pushmeet Kohli Joshua B Tenenbaum
references.title: Neural- Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
references.venue: In Advances in Neural Information Processing Systems (NIPS),
references.authors: Zhou Yu Jun Yu Yuhao Cui Dacheng Tao Qi Tian
references.title: Deep modular co-attention networks for visual question answering
references.venue: In CVPR,
references.authors: C. Zhang S. Bengio M. Hardt B. Recht O. Vinyals
references.title: Understanding deep learning requires rethinking generalization
references.venue: In ICLR,
sections.heading: 1 Introduction
sections.text: Reasoning over images is the main goal of Visual Question Anwering (VQA), a task where a model is asked to answer questions over images. This problem is a test bed for the creation of agents capable of high-level reasoning, as it involves multi-modal and high-dimensional data as well as complex decision functions requiring latent representations and multiple hops. State-of-the-art models are notorious for leveraging dataset biases and shortcuts in learning rather than performing reasoning, leading to lack of generalization, as evidenced by extensive recent work on bias oriented benchmarks for vision-and-language reasoning [1, 20, 21, 31]. Even large-scale semi-supervised pre-training methods, which successfully managed to increase overall VQA performance, still struggle to address questions whose answers are rare given a context [20].
  It has been recently shown that reasoning patterns emerge in attention layers of a SOTA VQA model when trained on perfect (oracle) visual inputs, which provides evidence that deep neural networks can learn to reason, when training conditions are favorable enough [21]. In particular, uncertainty and noise in visual inputs seems to be a major cause for shortcut learning in VQA. While this kind of methods provide strong empirical results and insights on the bottlenecks in problems involving learning to reason, they still suffer from significant loss in reasoning capabilities during the transfer phase, when the model is required to adapt from perfectly clean visual input to the noisy input it will encounter after deployment. We conjecture, that reasoning on noisy data involves additional functional components not necessary in the clean case due to different types of domain shifts: (1) a presence shift, caused by imperfect object detectors, leading to missing visual objects necessary for
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  reasoning, or to multiple (duplicate) detections; (2) an appearance shift causing variations in object embeddings (descriptors) for the same class of objects due to different appearance.
  In this paper, we propose a new method for transferring reasoning patterns from models learned on perfect visual input to models trained on noisy visual representations. Key to the success is a regularization term minimizing loss of the reasoning capabilities during transfer. In particular, we address this problem through program prediction as an additional auxiliary loss, i.e. supervision of the sequence of reasoning operations along with their textual and/or visual arguments. To maintain a strong link between the learned function and its objective during the knowledge transfer phase, when inputs are switched from clean oracle inputs to noisy input, the neural model is required to continue to predict complex reasoning programs from different types of inputs.
  As a second justification, we claim that program supervision in itself leads to a simpler learning problem, as the underlying reasoning function is decomposed into a set of tasks, each of which is easier to learn individually than the full joint decision function. We backup this claim through a theoretical analysis showing decreased sample complexity under mild hypotheses.
  As a summary, we present the following contributions: (i) we propose a new program supervision module added on top of vision-language transformer models; (ii) we provide a theoretical analysis of the benefit of supervising program prediction in VQA deriving bounds on sample complexity; (iii) we experimentally demonstrate the efficiency of program supervision and show that it increases VQA performance on both in- and out-of-distribution sets, even when combined with BERT-like pre-training [30, 10], and that it improves the quality of oracle transfer initially proposed by [21].
sections.heading: 2 Related Work
sections.text: Transformers in VQA — VQA as a task was introduced in various datasets [4, 12, 16], including GQA [15] which is automatically-generated from real-world images. This growing amount of diverse datasets has been accompanied by the development of more and more sophisticated VQA models. While their exhaustive survey is out of the scope of this paper, one can mention some foundational categories of approaches, e.g. those based on object-level attention [2] and tensor decomposition [5]. In this work, we focus on VQA models which are based on Transformers [33], due to their wide adoption and their impressive results in several tasks (including VQA). In particular, we rely on the combination of Transformers with a large-scale BERT [10]-like pretraining which was shown to be beneficial for VQA in recent works [19, 30]. More recently, [21] focused on the study of so-called reasoning patterns in such Transformer-based VQA models. The authors analyzed how various VQA tasks are encoded in different attention heads, by applying an energy-based analysis inspired by [24]. The analysis was performed using a perfect-sighted oracle Transformer model (which is trained with near-to-perfect visual information, and thus, is much less prone to exploit dataset biases) in order to identify which patterns lead to better reasoning. Then, the authors showed that these reasoning patterns can be transferred from the oracle to a Transformer-based VQA model, thus improving both overall accuracy and accuracy on infrequent answers. In this work, we argue that using program prediction as an additional supervision signal is a catalyst for the transfer of these reasoning patterns.
  Biases and shortcut learning in VQA — In addition to the popular VQA datasets mentioned above, other benchmarks were proposed to evaluate specific reasoning capabilities of VQA systems. In particular, they address the issue of shortcut learning [11] in deep learning, where models learn decision rules which are ineffective when tested on a domain or distribution different from the training one. For instance, VQA-CP [1] explicitly inverts the answer distribution between train and test splits. To cope with recent criticisms regarding these evaluations [31, 28], the GQA-OOD dataset [18] introduced a new split of GQA focusing on rare (Out-Of-Distribution / OOD) question-answer pairs, and showed that many VQA models strongly rely on dataset biases. Following this work, the VQA-CE dataset [8] introduced a new evaluation approach related to the VQA v2 dataset. By studying the multi-modal shortcuts in the training set and mining some trivial predictive rules (e.g. co-occurrences of words and visual objects), the evaluation set is generated using questions where these mined rules lead to incorrect answers. As for GQA-OOD, this dataset demonstrated that SOTA models do not perform well when they can’t rely on shortcuts, even models which use bias-reduction techniques. Based on these benchmarks, several methods have been conceived to reduce shortcut learning in VQA (see [31, 29] for a comprehensive study of the different techniques). However none of them achieve significant performance improvement on GQA-OOD [18] or VQA-CE [8].
  Connections with symbolic VQA — Our work is also related to neuro-symbolic reasoning [35, 23] and neural module networks (NMN) [3, 13], which generally encode a set of pre-defined functions into unique neural modules, then dynamically compose them to execute question-related programs. Several improvements of standard NMNs have been proposed to make them end-to-end trainable through reinforcement learning [17] or, more recently, to enhance their scalability and generalizability [6]. However, the common point of all these works is that they are generally based on the prediction of reasoning programs whose elementary functions are learned jointly with program prediction itself, through program supervision. In contrast to this work, our approach uses program supervision to enrich its internal representations instead of inferring program’s execution.
  Measuring complexity of learning problems — and thus generalization, has been a goal of theoretical machine learning since the early days, with a large body of work based on PAC-Learning [32, 27]. Traditionally, bounds have been provided ignoring data distributions and focusing uniquely on hypothesis classes (network structures in neural network language), e.g. as measured by VCdimension. Surprising experimental results on training networks on random samples have seemingly contradicted learning theory [37], in particular Rademacher Complexity. Recently, building on statistics of gradient descent, bounds have been proposed which take into account data distributions, notably [26]. Algorithmic alignment between neural network structures and the decomposition of underlying reasoning functions has been studied in [34], with a focus on algorithms based on dynamic programming. Our theoretical contribution in section 4 builds on the latter two methodologies and extends this type of analysis to intermediate supervision of reasoning programs.
sections.heading: 3 Knowledge transfer and program supervision
sections.text: We propose a method for transferring reasoning patterns from models learned on perfect visual inputs to models trained on noisy visual representations. In the lines of oracle transfer [21], we first pre-train a model on ground-truth visual data and then fine-tune on standard visual embeddings extracted with an object detector. The underlying hypothesis is that the noise and uncertainty in visual input prevents the model from learning to reason and leads to learning shortcuts.
  To minimize the loss of reasoning capabilities during the transfer over the presence shift between oracle and noisy visual objects, we propose a regularization technique, which supervises the prediction of reasoning steps required to answer the question. We therefore assume the existence of the following ground truth annotation of reasoning programs. A given data sample consists of a sequence {qi} of input question word embeddings, a set {vi} of input visual objects, the ground truth answer class y∗ as well as the groundtruth reasoning program, which is structured as a tree involving operations and arguments. Operations {o∗i } are elements of a predefined set {choose color, filter size, ...}. The arguments of these operations may be taken from (i) all question words, (ii) all visual objects, (iii) all operations — when an operation takes as argument the result of another operation. Hence, arguments are annotated as many-to-many relationships. In the question “Is there a motorbike or a plane?”, for instance, the operation “or” depends on the result of the two operations checking the existence of a specific object in the image. This is denoted as aq∗ij ∈{0, 1} where a q∗ ij =1 means that
  operation i is associated with question word j as argument and, similarly, av∗ij =1 indicating a visual argument and ad∗ij =1 an operation result argument.
  We propose to apply the regularization on top of the VL-Transformer architecture proposed in [30], based on sequences of self- and cross-modality attention. For this purpose, we define a trainable module for program generation (program decoder), added to the output of the VL-Transformer model as shown in Fig. 2 — an adaptation to other architectures would be straightforward.
  Program decoder — In the lines of [6], the program decoder has been designed in a coarse-to-fine fashion. It first generates À a coarse sketch of the program consisting only of the operations, which are then Á refined by predicting textual and visual arguments and dependencies between operations.
  À Coarse: operation — this module only predicts the sequence of operations {oi}i∈[0,n−1] using a recurrent neural network (RNN / GRU) [7] variant, whose initial hidden state is initialized with the yCLS token embedding of the VQA transformer — the same embedding from which classically the final answer y is predicted, cf. Figure 2. Inference is stopped when the special “STOP” operation is predicted. At each GRU time step i, a new hidden state hi is computed, from which the operation oi is classified with a linear projection. It is supervised with a cross-entropy loss Lop = ∑ iLCE(oi,o∗i ).
  Á Fine: input arguments — the coarse program is then refined by predicting the operations’ arguments. We first deal with textual and visual arguments only. Affinity scores aqij between each operation’s hidden hi and token qj embeddings are computed with a 2-layer feed-forward network from concatenated embeddings. They represent the probability of the word qj to belong to the argument set of operation oi. Similar scores avij are computed for operations and visual objects. They are supervised with BCE losses Lqarg = ∑ ijLBCE(a q ij ,a q∗ ij ) and Lvarg = ∑ ijLBCE(avij ,av∗ij ).
  Fine: op arguments — next the dependencies are predicted, i.e. arguments which correspond to results of other operations, and which structure the program into a tree. We deal with these arguments differently, and compute the set of dependency arguments for each operation oi with another GRU, whose hidden state is initialized with the hidden state hi of the operation. The argument index adij is a linear projection of the hidden state and supervised with BCE: Ldep = ∑ ijLBCE(adij ,ad∗ij ).
  Program supervision — The coarse-to-fine program decoder is trained with the four additional losses weighted by hyperparameters α, β, γ, δ.
  L = Lvqa︸︷︷︸ VQA +α.Lop + β.Ldep + γ.Lqarg + δ.Lvarg︸ ︷︷ ︸ Program supervision , (1)
  Ground truth programs — We use ground truth information from the GQA [15] dataset, whose questions have been automatically generated from real images. Each sample contains a program describing the operations and arguments required to derive the answer for each question. However, the GT programs have been created for GT visual arguments (GT objects), which do not exactly
  match the visual input of an object detector used during training and inference [2]. We therefore construct a soft target, by computing intersection-over-union (IoU) between GT and detected objects.
  Oracle transfer — Our method uses program supervision to regularize knowledge transfer from a visual oracle to noisy input, as introduced in [21]. Oracle transfer consists in pretraining the VLTransformer model on ground-truth one-hot visual inputs before performing BERT-like pre-training. It offers training conditions which are more favorable for learning reasoning capabilities. We perform the following steps: (1) Oracle pre-training on GT visual input on the GQA dataset; (2) (optionally) BERT-like pre-training on data from GQA unbalanced; (3) Finetuning on the final VQA-objective on the GQA dataset. Each one of these steps are regularized using program supervision.
sections.heading: 4 Sample complexity of program supervision
sections.text: We provide a theoretical analysis indicating that the prediction and supervision of reasoning programs can improve learnability in vision and language reasoning under some assumptions. In what follows, we denote with g “true” (but unknown) underlying reasoning functions, and by f functions approximating them, implemented as neural networks. The goal is to learn a function g able to predict a distribution y over answer classes given an input question and an input image, see Fig 1a. While in the experimental part we use state-of-the-art tranformer based models, in this theoretical analysis, we consider a simplified model, which takes as input the two vectorial embeddings q and v corresponding to, respectively, the question and the visual information (image), for instance generated by a language model and a convolutional neural network, and produces answers y∗ = g(q,v).
  We restrict this analysis to two-layer MLPs, as they are easier to handle theoretically than modern attention based models. The reasoning function g is approximated by neural network f parametrized by a vector θ and which predicts output answers y = f(q,v, θ).
  Our analysis uses PAC-learning [32] and builds on recent results providing bounds on sample complexity taking into account the data distribution itself. We here briefly reproduce Theorem 3.5. from paper [34], which, as an extension of a result in [26], provides a bound for sample complexity of overparametrized MLPs with vectorial outputs, i.e. MLPs with sufficient capacity for learning a given task:
  Theorem 4.1 (Sample complexity for overparametrized MLPs). Let A be an overparametrized and randomly initialized two-layer MLP trained with gradient descent for a sufficient number of iterations. Suppose g : Rd → Rm with components g(x)(i) = ∑ j α (i) j (β (i)T j x) p (i) j , where β(i)j ∈ Rd, α(i) ∈ R, and p(i)j = 1 or p (i) j = 2l, l ∈ N+. The sample complexity CA(g, , δ) is
  CA(g, , δ) = O maxi∑j p(i)j |α(i)j |·||β(i)j ||p(i)j2 + log(mδ ) ( /m)2  (2) We use the following Ansatz: since each possible input question requires a potentially different form of reasoning over the visual content, our analysis is based on the following assumption.
  Assumption 1. The unknown reasoning function g() is a mixture model which decomposes as follows y∗ = ∑ r πrhr = ∑ r πrgr(v), (3)
  where the different mixture components r correspond to different forms of reasoning related to different questions. The mixture components can reason on the visual input only, and the mixture weights are determined by the question q, i.e. the weights π depend on the question q, e.g. π = gπ(q).
  We call gπ(.) the reasoning mode estimator. One hypothesis underlying this analysis is that learning to predict reasoning programs allows the model to more easily decompose into this form (3), i.e. that the network structure closely mimicks this decomposition, as information on the different reasoning modes r is likely to be available in the activations of intermediate layers, cf. Figure 1. This will be formalized in assumption 3 and justified further below.
  Considering the supposed “true” reasoning function y∗ = g(q,v) and its decomposition given in (3), we suppose that each individual reasoning module gr can be approximated with a multi-variate
  polynomial, in particular each component h(i)r of the vector hr, as h(i)r = gr(v) = ∑ j α (i) r,j(β (i)T r,j v) p (i) r,j with parameters ω = { α (i) r,j , β (i) r,j , p (i) r,j . }
  (4)
  A trivial lower bound on the complexity of the reasoning mode estimator gπ(.) is the complexity of the identity function, which is obtained in the highly unlikely case where the question embeddings q contain the 1-in-K encoding of the choice of reasoning mode r. We adopt a more realistic case as the following assumption. Assumption 2. The input question embeddings q are separated into clusters according to reasoning modes r, such that the underlying reasoning mode estimator gπ can be realized as a NN classifier with dot-product similarity in this embedding space.
  Under this assumption, the reasoning mode estimator can be expressed as a generalized linear model, i.e. a linear function followed by a soft-max σ,
  π = gπ(q) = σ ([ γT0 q, γ T 1 q, ... ]) , (5)
  where the different γr are the cluster centers of the different reasoning modes r in the question embedding space. As the softmax is a monotonic non-linear function, its removal will not decrease sample complexity1, and the complexity can be bounded by the logits πr = γTr q. Plugging this into (3) we obtain that each component y∗(i) of the answer is expressed as the following function:
  y∗(i) = ∑ r ( γTr q )∑ j α (i) r,j(β (i)T r,j v) p (i) r,j (6)
  We can reparametrize this function by concatenating the question q and the visual input v into a single input vector x, which are then masked by two different binary masks, which can be subsumed into the parameters γr and β (i) r,j , respectively, giving
  y∗(i) = ∑ r ∑ j (γTr x)α (i) r,j(β (i)T r,j x) p (i) r,j (7)
  Extending Theorem 3.5. from [34], we can give our main theoretical result as the sample complexity of this function expressed as the following theorem. Theorem 4.2 (Sample complexity for multi-mode reasoning functions). Let A be an overparametrized and randomly initialized two-layer MLP trained with gradient descent for a sufficient number of iterations. Suppose g : Rd → Rm with components g(x)(i) =∑ r ∑ j(γ T r x)α (i) r,j(β (i)T r,j x) p (i) r,j where γr ∈ Rd, β(i)r,j ∈ Rd, α (i) r,j ∈ R, and p (i) r,j = 1 or p (i) r,j = 2l, l ∈ N+. The sample complexity CA(g, , δ) is
  CA(g, , δ) = O maxi∑r∑j πp(i)r,j |α|·||γr||2·||βr,j ||p(i)r,j2 + log(m/δ) ( /m)2  . The proof of this theorem is given in the supplementary material (Appendix A).
  Theorem 4.2 provides the sample complexity of the reasoning function g() under classical training. In the case of program supervision, our analysis is based on the following assumption (see also Fig. 1b): Assumption 3. Supervising reasoning programs encodes the choice of reasoning modes r into the hidden activations of the network f . Therefore, learning is separated into several different processes,
  (a) learning of the reasoning mode estimator gπ() approximated as a network branch fπ() connected to the program output;
  (b) learning of the the different reasoning modules gr() approximated as network branches fr() connected to the different answer classes yr; each one of these modules is learned independently.
  1In principle, there should exist special degenerate cases, where an additional softmax could reduce sample complexity; however, in our case it is applied to a linear function and thus generates a non-linear function.
  We justify Assumption 3.a through supervision directly, which separates gπ() from the rest of the reasoning process. We justify Assumption 3.b by the fact, that different reasoning modes r will lead to different hidden activations of the network. Later layers will therefore see different inputs for different modes r, and selector neurons can identify responsible inputs for each branch fr(), effectively switching off irrelevant input.
  We can see that these complexities are lower than the sample complexity of the full reasoning function given in theorem 4.2, since for a given combination of i, r, j, the term ||γr||2·||βr,j || p (i) r,j 2 dominates the corresponding term ||βr,j || p (i) r,j
  2 . Recalling that the vectors γ correspond to the cluster centers of reasoning modes in language embedding space. Under the assumption that the embeddings q have been created with batch normalization, a standard technique in deep learning, each value γ(i)r follows a normal distribution N (0, 1). Dropping indices i, r, j to ease notation, we can then compare the expectation of the term ||γ||2·||β||p2 over the distribution of γ and derive the following relationship:
  Eγ(i)∼N(0,1)||γ||2·||β|| p 2 = C||β|| p 2 =
  √ 2 Γ(m2 + 1 2 )
  Γ(m2 ) ||β||p2 (8)
  where Γ is the Gamma special function and m is the dimension of the language embedding γ. We provide a proof for this equality in the supplementary material (Appendix A).
  Discussion and validity of our claims — the difference in sample complexity is determined by the factor C in equation (8), which monotonically grows with the size of the embedding space m, which is typically in the hundreds. For the order of m=512 to m=768 used for state-of-the-art LXMERT models [30], complexity grows by a factor of around ∼20. We would like to point out that this analysis very probably under-estimates the difference in complexity, as the difference highly depends on the complexity of the reasoning estimator π, which we have simplified as a linear function in equation (5). Taking into account just the necessary soft-max alone would probably better appreciate the difference in complexity between the two methods, which we leave for future work. Our analysis is also based on several assumptions, among which is the simplified model (an over-parametrized MLP instead of an attention based network), as well as assumptions of Theorem 4.2 from [34] and [26], on which our analysis is based. Lastly, we would like to comment on the fact that we compare two different bounds: (i) the one on sample complexity for learning the full multi-modal reasoning given in Theorem 4.2, and (ii) the one for learning a single reasoning mode given by Theorem 4.1. While comparing bounds does not provide definitive answers on the order of models, both bounds have been derived by the same algebraic manipulations and we claim that they are comparable.
sections.heading: 5 Experimental results
sections.text: Setup + architecture: — we perform our experiments with a compact version of the VisionLanguage (VL)-Tansformer used in [30] (cf. Fig. 2), with a hidden embedding size of d=128 and h=4 heads per layer (only 26M trainable parameters). Dataset: our models are trained on the balanced GQA [15] training set (∼1M question-answer pairs). However, LXMERT pretraining is done on the unbalanced training set (∼15M question-answer pairs). The latter contains more questions and programs, but the same number of images (∼100K images). Evaluation: is performed on GQA [15]
  and GQA-OOD [18] test sets. GQA is a dataset with question-answer pairs automatically generated from real images, and is particularly well suited for evaluating a large variety of reasoning skills. GQA-OOD is a benchmark dedicated to the out-of-domain VQA evaluation, and gives information on the generalization capabilities of the model. Hyper-parameters are selected either on the testdev (for GQA) or validation (for GQA-OOD) sets. When specified (with ±) we provide the average accuracy and standard deviation computed on three runs with different random seeds. Visual input: following [2], we use bottom-up visual features extracted using a pre-trained object detector (we keep its parameters frozen during the training). If not specified, we use faster-RCNN [25] with 36 objects per-images. In addition to that, we experiment with designed objects, and also with the VinVL [38] features which (unlike faster-RCNN ones) are conceived specifically for vision-language tasks.
  Program supervision improves visual reasoning — Tab. 1 reports the effectiveness of program prediction when combined with oracle and BERT-like pretraining on the GQA dataset and corroborates the results found in the theoretical analysis. In addition, when using both program supervision and LXMERT [30] but without oracle transfer, we achieve an accuracy of 58.8 on the testdev set of GQA. This is lower than oracle transfer’s accuracy, demonstrating the complementary of the two methods. We note that the majority of the gain is achieved on the more challenging open questions. In addition, results on GQA-OOD (acc-tail and acc-head) suggest that the gains are obtained in, both, out- and in-distribution settings. However, as already observed in [21], LXMERT pre-training tends to decrease the acc-tail gains brought by oracle transfer plus program supervision. We evaluate the program prediction performance by measuring the area under the ROC curve (AUC) on the visual argument prediction with an IoU treshold of 23=0.66. Models (c) and (e) achieve, respectively, 97.1 and 96.4 AUC scores, demonstrating the effectiveness of the program decoder.
  Program supervision decreases sample complexity — as shown in Fig. 3. We vary the amount of training data from 5% to 100% and observe that adding program supervision allows to reach an accuracy similar to the baseline while using less data (no oracle transfer, no LXMERT pretraining).
  Visual arguments are the key — We study the impact of different types of program supervision in Tab. 2. We can see the importance of supervising arguments, in (4) and (5). The supervision of visual arguments (5) contributes most to the gain in performance, again corroborating that visual uncertainty is the main bottleneck for reasoning.
  Program supervision enhances cross-modal interactions — In Tab. 3, we study how the inputs of the program prediction module influence the VQA accuracy. In particular, we test two settings: (7) uni-modal, where the programs are predicted from the vision and language em-
  beddings right after the uni-modal layers (language and vision only in Fig. 2); and (8) cross-modal, where the programs are predicted after the cross-modal layers (as shown in Fig. 2). We observe that, contrary to the latter, the former does not improve the baseline ((7) vs (6) in Tab. 3). This highlights the fact that the program supervision mainly impacts the operations in the cross modal layers, where the most complex reasoning operations are performed.
  Program supervision allows to take advantage of the improved visual inputs — We analyse the impact of using our method with a better input image representation. Increasing the number of objects from 36 to 100 per image ((g) and (h) in Tab. 4), allows to further increase the gains brought by our method. On the contrary, the score of the baseline model remains unchanged, showing that the program supervision allows to take advantage of a bigger number of object proposals. Similarly, replacing the faster-RCNN features by the more recent and more accurate VinVL ones ((i)-(l) in Tab. 4) results in better performances.
  Comparison with SOTA — We report in Tab. 5 the results obtained by our approach compared to the current SOTA on the GQA and GQA-OOD datasets. In order to ensure a fair comparison, we also provide, for each method, information regarding the amount of data (images and sentences) used during training. As shown in Tab. 5, our approach compares favourably with SOTA since it obtains the second best accuracy (with a 0.2 points gap) on the GQA test-std set among the approaches which not use extra training data. The results also remain competitive when comparing to the OSCAR+VinVL [38], while being trained with 50 times less images. On GQA-OOD, our approach obtains the second best acc-tail score (and the best acc-head one) with a much less complex architecture than current SOTA (26M vs 212M trainable parameters compared to LXMERT [30]).
sections.heading: 6 Conclusion
sections.text: We have demonstrated that it is possible to improve the reasoning abilities of VQA models when providing additional supervision of program annotations. In particular, our method is designed to improve the transfer of reasoning patterns from models learned on perfect visual input to models trained on noisy visual representations. Our experiments are supported by a theoretical analysis, demonstrating that program supervision can decrease sample complexity under reasonable hypothesis. The proposed method relies on the availability of reasoning program annotations, which are costly to annotate, especially when dealing with human generated questions. Recent work has already managed to gather such kind of annotations [9]. The next step will be to extend the method to configurations where the program annotation is rare or incomplete. Another promising research path could be explicitly conditioning the VQA system’s answer by the program prediction module, following the recent work of Chen et al. [6], instead of using it as catalyst for knowledge transfer.
  Acknowledgment — C. Wolf acknowledges support from ANR through grant “Remember” (ANR20-CHIA-0018).
title: Supervising the Transfer of Reasoning Patterns in VQA