Neural module networks (NMNs) are a popular approach for modeling compositionality: they achieve high accuracy when applied to problems in language and vision, while reflect-ing the compositional structure of the problem in the network architecture. However, prior work implicitly assumed that the structure of the network modules, describing the abstract reasoning process, provides a faithful explanation of the model's reasoning; that is, that all modules perform their intended behaviour. In this work, we propose and conduct a systematic evaluation of the intermediate outputs of NMNs on NLVR2 and DROP, two datasets which require composing multiple reasoning steps. We find that the intermediate outputs differ from the expected output, illustrating that the network structure does not provide a faithful explanation of model behaviour. To remedy that, we train the model with auxiliary supervision and propose particular choices for module architecture that yield much better faithfulness, at a minimal cost to accuracy. Models that can read text and reason about it in a particular context (such as an image, a paragraph, or a table) have been recently gaining increased attention, leading to the creation of multiple datasets that require reasoning in both the visual and textual domain (Johnson et al., 2016; Suhr et al., 2017; Talmor and Berant, 2018; Yang et al., 2018a; Suhr et al., 2019; Hudson and Manning, 2019; Dua et al., 2019). Consider the example in Figure 1 from NLVR2: a model must understand the compositional sentence in order to then ground dogs in the input, count those that are black and verify that the count of all dogs in the image is equal to the number of black dogs. Both models that assume an intermediate structure (Andreas et al., 2016; Jiang and Bansal, 2019) and models without such structure (Tan and Bansal, 2019; Hu et al., 2019; Min et al., 2019) have been proposed for these reasoning problems. While good performance can be obtained without a structured representation, an advantage of structured approaches is that the reasoning process in such approaches is more interpretable . For example, a structured model can explicitly denote that there are two dogs in the image, but that one of them is not black . Such interpretability improves our sci-entific understanding, aids in model development, and improves overall trust in a model. Neural module networks (NMNs; Andreas et al., 2016) parse an input utterance into an executable program composed of learnable modules that are designed to perform atomic reasoning tasks and can be composed to perform complex reasoning against an unstructured context. NMNs are appealing since their output is interpretable; they provide a logical meaning representation of the utterance and also the outputs of the intermediate steps (modules) to reach the final answer. However, because module parameters are typically learned from end-task supervision only, it is possible that the program will not be a faithful explanation of the behaviour of the model (Ross et al., 2017; Wiegreffe and Pinter, 2019), i.e., the model will solve the task by executing modules according to the program structure, but the modules will not perform the reasoning steps as intended . For example, in Figure 1, a basic NMN predicts the correct answer False , but incorrectly predicts the output of the find [ dogs ] operation. It does not correctly locate one of the dogs in the image because two of the reasoning steps ( find and filter ) are collapsed into one module ( find ). This behavior of the find module is not faithful to its intended reasoning operation; a human reading the program would expect find [ dogs ] to locate all dogs. Such unfaithful module behaviour yields an unfaithful explanation of the model behaviour. Unfaithful behaviour of modules, such as multiple reasoning steps collapsing into one, are undesirable in terms of interpretability; when a model fails to answer some question correctly, it is hard to tell which modules are the sources of error. While recent work (Yang et al., 2018b; Jiang and Bansal, 2019) has shown that one can obtain good performance when using NMNs, the accuracy of individual module outputs was mostly evaluated through qualitative analysis, rather than systematically evaluating the intermediate outputs of each module. We provide three primary contributions regarding faithfulness in NMNs. First, we propose the concept of module-wise faithfulness  a systematic evaluation of individual module performance in NMNs that judges whether they have learned their intended operations, and define metrics to quantify this for both visual and textual reasoning (  3). Empirically, we show on both NLVR2 (Suhr et al., 2019) and DROP (Dua et al., 2019) that training a NMN using end-task supervision, even using gold programs , does not yield module-wise faithfulness, i.e., the modules do not perform their intended reasoning task. Second, we provide strategies for improving module-wise faithfulness in NMNs (  4). Specifically, (a) we demonstrate how module architecture affects faithfulness (  4.1), (b) propose supervising module outputs with either a proxy task or heuristically generated data (  4.2), and (c) show that providing modules with uncon-texualized token representations improves faithfulness (  4.3). Figure 1 shows an example where our approach ( Faithful-NMN ) results in expected module outputs as compared to the Basic-NMN . Last, we collect human-annotated intermediate outputs for 536 examples in NLVR2 and for 215 examples in DROP to measure the module-wise faithfulness of models, and publicly release them for future work. Our code and data are available at https://github.com/allenai/faithful-nmn . Overview Neural module networks (NMNs; Andreas et al., 2016) are a class of models that map a natural language utterance into an executable program, composed of learnable modules that can be executed against a given context (im-ages, text, etc.), to produce the utterance's denotation (truth value in NLVR2, or a text answer in DROP). Modules are designed to solve atomic reasoning tasks and can be composed to perform complex reasoning. For example, in Figure 1, the utterance All the dogs are black is mapped to the program equal(count(find[ dogs ]), count(filter[ black ](find[ dogs ]))) . The find module is expected to find all dogs in the image and the filter module is expected to output only the black ones from its input. Figure 2 shows two other example programs with the expected output of each module in the program. A NMN has two main components: (1) parser, which maps the utterance into an executable program; and (2) executor, which executes the program against the context to produce the denotation. In our setup, programs are always trees where each tree node is a module. In this work, we focus on the executor, and specifically the faithfulness of module execution. We examine NMNs for both text and images, and describe their modules next. In this task, given two images and a sentence that describes the images, the model should output True iff the sentence correctly describes the images. We base our model, the Visual-NMN, on LXMERT (Tan and Bansal, 2019), which takes as input the sentence x and raw pixels, uses Faster R-CNN (Ren et al., 2015) to propose a set of bounding boxes, B , that cover the objects in the image, and passes the tokens of x and the bounding boxes through a Transformer (Vaswani et al., 2017), encoding the interaction between both modalities. This produces a contextualized representation t  R | x | h for each one of the tokens, and a representation v  R |B| h for each one of the bounding boxes, for a given hidden dimension h . We provide a full list of modules and their implementation in Appendix A. Broadly, modules take as input representations of utterance tokens through an utterance attention mechanism (Hu et al., 2017), i.e., whenever the parser outputs a module, it also predicts a distribution over the utterance tokens ( p 1 , . . . , p | x | ) , and the module takes as input (cid:80) | x | i =1 p i t i , where t i is the hidden representation of token i . In addition, modules produce as output (and take as input) vectors p  [0 , 1] |B| , indicating for each bounding box the probability that it should be output by the module (Mao et al., 2019). For example, in the program filter [ black ]( find [ dog ]), the find module takes the word dog' (using utterance attention , which puts all probability mass on the word dog'), and outputs a probability vector p  [0 , 1] |B| , where ideally all bounding boxes corresponding to dogs have high probability. Then, the filter module takes p as input as well as the word black', and is meant to output high probabilities for bounding boxes with black dogs'. For the Visual-NMN we do not use a parser, but rely on a collected set of gold programs (including gold utterance attention ), as described in  5. We will see that despite this advantageous setup, a basic NMN does not produce interpretable outputs. Our Text-NMN is used to answer questions in the DROP dataset and uses the modules as designed for DROP in prior work (Gupta et al., 2020) along with three new modules we define in this work. The modules introduced in Gupta et al. (2020) and used as is in our Text-NMN are find , filter , relocate , count , find-num , find-date , find-max-num , find-min-num , num-compare and date-compare . All these modules are probabilistic and produce, as output, a distribution over the relevant support. For example, find outputs a distribution over the passage tokens and find-num outputs a distribution over the numbers in the passage. We extend their model and introduce additional modules; addition and subtraction to add or subtract passage numbers, and extract-answer which directly predicts an answer span from the representations of passage tokens without any explicit compositional reasoning. We use BERT-base (Devlin et al., 2019) to encode the input question and passage. The Text-NMN does not have access to gold programs, and thus we implement a parser as an encoder-decoder model with attention similar to Krishnamurthy et al. (2017), which takes the utterance as input, and outputs a linearized abstract syntax tree of the predicted program. Neural module networks (NMNs) facilitate interpretability of their predictions via the reasoning steps in the structured program and providing the outputs of those intermediate steps during execution. For example, in Figure 2, all reasoning steps taken by both the Visual-NMN and Text-NMN can be discerned from the program and the intermediate module outputs. However, because module parameters are learned from an end-task, there is no guarantee that the modules will learn to perform their intended reasoning operation. In such a scenario, when modules do not perform their intended reasoning, the program is no longer a faithful explanation of the model behavior since it is not possible to reliably predict the outputs of the intermediate reasoning steps given the program. Work on NMNs thus far (Yang et al., 2018b; Jiang and Bansal, 2019) has overlooked systematically evaluating faithfulness, performing only qualitative analysis of intermediate outputs. We introduce the concept of module-wise faithfulness aimed at evaluating whether each module has correctly learned its intended operation by judging the correctness of its outputs in a trained NMN. For example, in Figure 2 (top), a model would be judged module-wise faithful if the outputs of all the modules, find , relocate , and with relation , are correct  i.e. similar to the outputs that a human would expect. We provide gold programs when evaluating faithfulness, to not conflate faithfulness with parser accuracy. Modules in Visual-NMN provide for each bounding box a probability for whether it should be a module output. To evaluate intermediate outputs, we sampled examples from the development set, and annotated gold bounding boxes for each instance of find , filter , with-relation and relocate . The annotator draws the correct bounding-boxes for each module in the gold program, similar to the output in Figure 2 (top). A module of a faithful model should assign high probability to bounding-boxes that are aligned with the annotated bounding boxes and low probabilities to other boxes. Since the annotated bounding boxes do not align perfectly with the model's bounding boxes, our evaluation must first induce an alignment. We consider two bounding boxes as aligned if the intersection-over-union (IOU) between them exceeds a pre-defined threshold T = 0 . 5 . Note that it is possible for an annotated bounding box to be aligned with several proposed bounding boxes and vice versa. Next, we consider an annotated bounding box BA as matched w.r.t a module output if BA is aligned with a proposed bounding box BP , and BP is assigned by the module a probability > 0 . 5 . Similarly, we consider a proposed bounding box BP as matched if BP is assigned by the module a probability > 0 . 5 and is aligned with some annotated bounding box BA . We compute precision and recall for each module type (e.g. find ) in a particular example by considering all instances of the module in that example. We define precision as the ratio between the number of matched proposed bounding boxes and the number of proposed bounding boxes assigned a probability of more than 0.5. We define recall as the ratio between the number of matched annotated bounding boxes and the total number of annotated bounding boxes. 1 F 1 is the harmonic mean of precision and recall. Similarly, we compute an overall precision, recall, and F 1 score for an example by considering all instances of all module types in that example. The final score is an average over all examples. Please see Appendix B.2 for further discussion on this averaging. Each module in Text-NMN produces a distribution over passage tokens (  2.2) which is a soft distributed representation for the selected spans. To measure module-wise faithfulness in Text-NMN, we obtain annotations for the set of spans that should be output by each module in the gold program (as seen in Figure 2 (bottom)) Ideally, all modules ( find , filter , etc.) should predict high probability for tokens that appear in the gold spans and zero probability for other tokens. To measure a module output's correctness, we use a metric akin to cross-entropy loss to measure the deviation of the predicted module output p att from the gold spans S = [ s 1 , . . . , s N ] . Here each span s i = ( t i s , t i e ) is annotated as the start and end tokens. Faithfulness of a module is measured by: I =  (cid:80) Ni =1 (cid:32) log (cid:80) t i e j = t i s p j att (cid:33) . Lower cross-entropy corresponds to better faithfulness of a module. Module-wise faithfulness is affected by various factors: the choice of modules and their implementation (  4.1), use of auxiliary supervision (  4.2), and the use of contextual utterance embeddings (  4.3). We discuss ways of improving faithfulness of NMNs across these dimensions. Visual reasoning The count module always appears in NLVR2 as one of the top-level modules (see Figures 1 and 2). 2 We now discuss how its architecture affects faithfulness. Consider the program, count(filter[ black ](find[ dogs ])) . Its gold denotation (correct count value) would provide minimal feedback using which the descendant modules in the program tree, such as filter and find , need to learn their intended behavior. However, if count is implemented as an expressive neural network, it might learn to perform tasks designated for find and filter , hurting faithfulness. Thus, an architecture that allows counting, but also encourages descendant modules to learn their intended behaviour through backpropagation, is desirable. We discuss three possible count architectures, which take as input the bounding box probability vector p  [0 , 1] |B| and the visual features v  R |B| h . Layer-count module is motivated by the count architecture of Hu et al. (2017), which uses a linear projection from image attention, followed by a softmax. This architecture explicitly uses the visual features, v , giving it greater expressivity compared to simpler methods. First we compute p  v , the weighted sum of the visual representations, based on their probabilities, and then output a scalar count using: FF 1 ( LayerNorm ( FF 2 ( p  v )) , where FF 1 and FF 2 are feed-forward networks, and the activation function of FF 1 is ReLU in order to output positive numbers only. As discussed, since this implementation has access to the visual features of the bounding boxes, it can learn to perform certain tasks itself, without providing proper feedback to descendant modules. We show in  5 this indeed hurts faithfulness. Sum-count module on the other extreme, ignores v , and simply computes the sum (cid:80) |B| i =1 p i . Be-2 Top-level modules are Boolean quantifiers, such as number comparisons like equal (which require count ) or exist . We implement exist using a call to count and greater-equal (see Appendix A), so count always occurs in the program. ing parameter-less, this architecture provides direct feedback to descendant modules on how to change their output to produce better probabilities. However, such a simple functional-form ignores the fact that bounding boxes are overlapping, which might lead to over-counting objects. In addition, we would want count to ignore boxes with low probability. For example, if filter predicts a 5% probability for 20 different bounding boxes, we would not want the output of count to be 1 . 0 . Graph-count module (Zhang et al., 2018) is a mid-dle ground between both approaches the nave Sum-Count and the flexible Layer-Count . Like Sum-Count , it does not use visual features, but learns to ignore overlapping and low-confidence bounding boxes while introducing only a minimal number of parameters (less than 300 ). It does so by treating each bounding box as a node in a graph, and then learning to prune edges and cluster nodes based on the amount of overlap between their bounding boxes (see paper for further details). Because this is a light-weight implementation that does not access visual features, proper feedback from the module can propagate to its descendants, encouraging them to produce better predictions. Textual reasoning In the context of Text-NMN (on DROP), we study the effect of several modules on interpretability. First, we introduce an extract-answer module. This module bypasses all compositional reasoning and directly predicts an answer from the input contextualized representations. This has potential to improve performance, in cases where a question describes reasoning that cannot be captured by pre-defined modules, in which case the program can be the extract-answer module only. However, introducing extract-answer adversely affects interpretability and learning of other modules, specifically in the absence of gold programs. First, extract-answer does not provide any interpretability. Second, whenever the parser predicts the extract-answer module, the parameters of the more interpretable modules are not trained. Moreover, the parameters of the encoder are trained to perform reasoning internally in a noninterpretable manner. We study the interpretability vs. performance trade-off by training Text-NMN with and without extract-answer . Second, consider the program find-max-num(find[ touchdown ]) that aims to find the longest touchdown . find-max-num should sort spans by their value and return the maximal one; if we remove find-max-num , the program would reduce to find[ touchdown ] , and the find module would have to select the longest touchdown rather than all touchdowns, following the true denotation. More generally, omitting atomic reasoning modules pushes other modules to compensate and perform complex tasks that were not intended for them, hurting faithfulness. To study this, we train Text-NMN by removing sorting and comparison modules (e.g., find-max-num and num-compare ), and evaluate how this affects module-wise interpretability. As explained, given end-task supervision only, modules may not act as intended, since their parameters are only trained for minimizing the end-task loss. Thus, a straightforward way to improve interpretability is to train modules with additional atomic-task supervision. Visual reasoning For Visual-NMN, we pre-train find and filter modules with explicit intermediate supervision, obtained from the GQA balanced dataset (Hudson and Manning, 2019). Note that this supervision is used only during pre-training  we do not assume we have full-supervision for the actual task at hand. GQA questions are annotated by gold programs; we focus on exist questions that use find and filter modules only, such as Are there any red cars? . Given gold annotations from Visual Genome (Kr-ishna et al., 2017), we can compute a label for each of the bounding boxes proposed by Faster-RCNN. We label a proposed bounding box as positive' if its IOU with a gold bounding box is > 0 . 75 , and negative' if it is < 0 . 25 . We then train on GQA examples, minimizing both the usual denotation loss, as well as an auxiliary loss for each instance of find and filter , which is binary cross entropy for the labeled boxes. This loss rewards high probabilities for positive' bounding boxes and low probabilities for negative' ones. Textual reasoning Prior work (Gupta et al., 2020) proposed heuristic methods to extract supervision for the find-num and find-date modules in DROP. On top of the end-to-end objective, they use an auxiliary objective that encourages these modules to output the gold numbers and dates according to the heuristic supervision. They show that supervising intermediate module outputs helps improve model performance. In this work, we evaluate the effect of such supervision on the faithfulness of both the supervised modules, as well as other modules that are trained jointly. The goal of decomposing reasoning into multiples steps, each focusing on different parts of the utterance, is at odds with the widespread use of contextualized representations such as BERT or LXMERT. While the utterance attention is meant to capture information only from tokens relevant for the module's reasoning, contextualized token representations carry global information. For example, consider the program filter[ red ](find[ car ]) for the phrase red car . Even if find attends only to the token car , its representation might also express the attribute red , so find might learn to find just red cars , rather than all cars , rendering the filter module useless, and harming faithfulness. To avoid such contextualiza-tion in Visual-NMN, we zero out the representations of tokens that are unattended, thus the input to the module is computed (with LXMERT) from the remaining tokens only. We first introduce the datasets used and the experimental setup for measuring faithfulness (  5.1). We demonstrate that training NMNs using end-task supervision only does not yield module-wise faithfulness both for visual and textual reasoning. We then show that the methods from  4 are crucial for achieving faithfulness and how different design choices affect it (  5.2). Finally, we qualitatively show examples of improved faithfulness and analyze possible reasons for errors (  5.3). Please see Appendix C for further detail about the experimental setups. Visual reasoning We automatically generate gold program annotations for 26 , 311 training set examples and for 5 , 772 development set examples from NLVR2. The input to this generation process is the set of crowdsourced question decompositions from the BREAK dataset (Wolfson et al., 2020). See Appendix C.1 for details. For module-wise faithfulness evaluation, 536 examples from the development set were annotated with the gold output for each module by experts. Table 1 : Faithfulness and accuracy on NLVR2. decont. refers to decontextualized word representations. Precision, recall, and F 1 are averages across examples, and thus F 1 is not the harmonic mean of the corresponding precision and recall. Table 2 : Faithfulness and performance scores for various NMNs on DROP. lower is better. min-max is average faithfulness of find-min-num and find-max-num ; find-arg of find-num and find-date . Textual reasoning We train Text-NMN on DROP, which is augmented with program supervision for 4 , 000 training questions collected heuristically as described in Gupta et al. (2020). The model is evaluated on the complete development set of DROP which does not contain any program supervision. Module-wise faithfulness is measured on 215 manually-labeled questions from the development set, which are annotated with gold programs and module outputs (passage spans). Visual reasoning Results are seen in Table 1. Accuracy for LXMERT, when trained and evaluated on the same subset of data, is 71.7%; slightly higher than NMNs, but without providing evidence for the compositional structure of the problem. For faithfulness, we measure an upper-bound on the faithfulness score. Recall that this score measures the similarity between module outputs and annotated outputs. Since module outputs are constrained by the bounding boxes proposed by Faster-RCNN (  2.1), while annotated boxes are not, perfect faithfulness could only be achieved by a model if there are suitable bounding boxes. Upper Bound shows the maximal faithfulness score conditioned on the proposed bounding boxes. We now compare the performance and faithfulness scores of the different components. When training our NMN with the most flexible count module, ( NMN w/ Layer-count ), an accuracy of 71 . 2% is achieved, a slight drop compared to LXMERT but with low faithfulness scores. Using Sum-count drops about 3% of performance, but increases faithfulness. Using Graph-count increases accuracy while faithfulness remains similar. Next, we analyze the effect of decontextualized word representations (abbreviated decont.) and pre-training. First, we observe that NMN w/ Graph-count + decont. increases faithfulness score to 0 . 33 F 1 at the expense of accuracy, which drops to 67 . 3% . Pre-training ( NMN w/ Graph-count + pretraining ) achieves higher faithfulness scores with a higher accuracy of 69 . 6% . Combining the two achieves the best faithfulness ( 0 . 47 F 1 ) with a minimal accuracy drop. We perform a paired permutation test to compare NMN w/ Graph-count + decont. + pretraining with NMN w/ Layer-count and find that the difference in F 1 is statistically significant ( p < 0 . 001 ). Please see Appendix D.1 for further details. Textual reasoning As seen in Table 2, when trained on DROP using question-program supervision, the model achieves 65 . 3 F 1 performance and a faithfulness score of 11 . 2 . When adding supervision for intermediate modules (  4.2), we find that the module-wise faithfulness score improves to 6 . 5 . Similar to Visual-NMN, this shows that supervising intermediate modules in a program leads to better faithfulness. To analyze how choice of modules affects faithfulness, we train without sorting and comparison modules ( find-max-num , num-compare , etc.). We find that while performance drops slightly, faithfulness deteriorates significantly to 8 . 4 , showing that modules that perform atomic reasoning are crucial for faithfulness. When trained without program supervision, removing extract-answer improves faithfulness ( 9 . 5  6 . 9 ) but at the cost of performance ( 63 . 5  60 . 8 F 1 ). This shows that such a black-box module encourages reasoning in an opaque manner, but can improve performance by overcoming the limitations of pre-defined modules. All improvements in faithfulness are significant as measured using paired permutation tests ( p < 0 . 001 ). Generalization A natural question is whether models that are more faithful also generalize better. We conducted a few experiments to see whether this is true for our models. For NLVR2, we performed (1) an experiment in which programs in training have length at most 7 , and programs at test time have length greater than 7 , (2) an experiment in which programs in training have at most 1 filter module and programs at test time have at least 2 filter modules, and (3) an experiment in which programs in training do not have both filter and with-relation modules in the same program, while each program in test has both modules. We compared three of our models  NMN w/ Layer-count , NMN w/ Sum-count , and NMN w/ Graph-count + decont. + pretraining . We did not observe that faithful models generalize better (in fact, the most unfaithful model tended to achieve the best generalization). To measure if faithful model behavior leads to better generalization in Text-NMN we conducted the following experiment. We selected the subset of data for which we have gold programs and split the data such that questions that require maximum and greater-than operations are present in the training data while questions that require computing minimum and less-than are in the test data. We train and test our model by providing gold-programs under two conditions, in the presence and absence of additional module supervision. We find that providing auxiliary module supervision (that leads to better module faithfulness; see above) also greatly helps in model generalization (perfor-mance increases from 32 . 3 F 1  78 . 3 F 1 ). Model comparisons We analyze outputs of different modules in Figure 3. Figures 3a, 3b show the output of find [ llamas ] when trained with contextualized and decontextualized word representations. With contextualized representations (3a), the find fails to select any of the llamas , presumably because it can observe the word eating , thus effectively searching for eating llamas , which are not in the image. Conversely, the decontextualized model correctly selects the boxes. Figure 3c shows that find outputs meaningless probabilities for most of the bounding boxes when trained with Layer-count , yet the count module produces the correct value (three). Figure 3d shows that find fails to predict all relevant spans when trained without sorting modules in Text-NMN. Error analysis We analyze cases where outputs were unfaithful. First, for visual reasoning, we notice that faithfulness scores are lower for long-tail objects. For example, for dogs , a frequent noun in NLVR2, the execution of find[ dogs ] yields an average faithfulness score of 0.71, while items such as roll of toilet paper , barbell and safety pin receive lower scores (0.22, 0.29 and 0.05 respectively; example for a failure case for safety pin in Fig. 3e). In addition, some objects are harder to annotate with a box ( water , grass , ground ) and therefore receive low scores. The issue of small objects can also explain the low scores of relocate . In the gold box annotations used for evaluation, the average areas for find , filter , with-relation , and relocate (as a fraction of the total image area) are 0 . 19 , 0 . 19 , 0 . 15 , and 0 . 07 , respectively. Evidently, relocate is executed with small objects that are harder to annotate ( tongue , spots , top of ), and indeed the upper-bound and model scores for relocate are lowest among the module types. NMNs were originally introduced for visual question answering and applied to datasets with syn-utt: In such prior work, module-wise faithfulness was mostly assessed via qualitative analysis of a few examples (Jiang and Bansal, 2019; Gupta et al., 2020). Yang et al. (2018b) did an evaluation where humans rated the clarity of the reasoning process and also tested whether humans could detect model failures based on module outputs. In contrast, we quantitatively measure each module's predicted output against the annotated gold outputs. A related systematic evaluation of interpretability in VQA was conducted by Trott et al. (2018). They evaluated the interpretability of their VQA counting model, where the interpretability score is given by the semantic similarity between the gold label for a bounding box and the relevant word(s) in the question. However, they studied only counting questions, which were also far less compositional than those in NLVR2 and DROP. Similar to the gold module output annotations that we provide and evaluate against, HOTPOTQA (Yang et al., 2018a) and COQA (Reddy et al., 2019) datasets include supporting facts or rationales for the answers to their questions, which can be used for both supervision and evaluation. In concurrent work, Jacovi and Goldberg (2020) recommend studying faithfulness on a scale rather than as a binary concept. Our evaluation method can be viewed as one example of this approach. We introduce the concept of module-wise faithfulness , a systematic evaluation of faithfulness in neural module networks (NMNs) for visual and textual reasoning. We show that nave training of NMNs does not produce faithful modules and propose several techniques to improve module-wise faithfulness in NMNs. We show how our approach leads to much higher module-wise faithfulness at a low cost to performance. We encourage future work to judge model interpretability using the proposed evaluation and publicly published annotations, and explore techniques for improving faithfulness and interpretability in compositional models. We thank members of UCI NLP, TAU NLP, and the AllenNLP teams as well as Daniel Khashabi for comments on earlier drafts of this paper. We also thank the anonymous reviewers for their comments. This research was partially supported by The Yandex Initiative for Machine Learning, the European Research Council (ERC) under the European Union Horizons 2020 research and innovation programme (grant ERC DELPHI 802800), funding by the ONR under Contract No. N00014-19-1-2620, and by sponsorship from the LwLL DARPA program under Contract No. FA8750-19-2-0201. This work was completed in partial fulfillment for the Ph.D degree of Ben Bogin.
abstain abstain objective objective objective other abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective result result abstain objective abstain abstain result method other abstain abstain abstain abstain abstain abstain abstain method abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain result result abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result result abstain abstain abstain abstain abstain result other abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other objective other other other method other abstain abstain objective result objective other other other other other other