summaries: In this paper, the authors propose to explain the behavior of black-box neural networks by approximating a top-level hidden layer (no non-linearities afterwards, except possibly softmax) as a linear combination of the hidden layers obtained from different examples. These examples may come from the training data, but it is not required. The contribution of each input feature is also evaluated using an extension of integrated gradient across multiple dimensions.

summaries: This paper introduces SimplEx, a method that approximate the hidden representation of a test example using the linear combination of hidden representations of examples in a pre-defined corpus. Further, the hidden representation of the test instance is associated with each input feature of these examples in the corpus. The paper demonstrates the quality of approximation of SimplEx (Sec 3.1, SimplEx outperforms kNN baselines) and demonstrate its practical utility in the experiment of predicting cancer mortality (Sec 3.2) by allowing model users to interpret model predictions and determine whether to trust the prediction. Detailed math derivation and additional experiments with MNIST and time series data are included in the appendix.

summaries: This paper introduces SimplEx, which is a method to post-hoc explain a model prediction by computing from a user-defined set of examples, how the set of examples decompose the prediction. For example, the clinical risk prediction of a patient can be explained by a set of other patients and how much each patient influenced the prediction.   Contributions -	A post-hoc explanation method, called SimplEx, were users can choose the set of examples from which the explanation should be constructed. This is advantageous when the training set is not available or a user wants customized explanations, e.g. a doctor wanting to understand a predictions in terms of other patients she/he knows. -	SimplEx combines the ideas of feature saliency and example-based explanations. For each example, the method computes the percentage of this example contributing to explaining the prediction as well as which features of this example were important. -	A latent-space generalization of Integrated Gradients [10] -	Experimental evaluation that show SimplEx to be more precise and robust compared to baselines.

summaries: The main contribution of this paper is SimplEx: a method that aims to explain a test prediction by decomposing it into the weighted sum of nearest neighbors (from a user-specified corpus) in the latent space. They claim this method is more precise and robust than existing explanation-by-example-comparison methods (namely, KNN-like explanations) and distinguish this method from (often gradient-based) feature saliency methods. The paper builds off work that try to explain model predictions that use comparisons in latent variable space (e.g., Concept Activation Vectors, Deep KNN). SimplEx works PROVIDED the model has a hidden layer that linearly maps to the output (this is the layer that is used for explaining predictions).   - This constraint is significant for accurate weighting of examples. Note that if this is not satisfied by the model (say for instance a softmax is added right before the final output), it is possible to consider an earlier layer that satisfies the linear-mapping constraint as the output  Note too that if the latent space spanned by the user uploaded corpus does not contain the latent representation of the test prediction of interest (it likely will not), the test's latent is projected into the space. This means that it is possible to use this method on any input, regardless of the corpus. We can then measure how good this approximation is by calculating the residual between the corpus span and the test latent.  We can then "interpret the shift in latent space as resulting from a shift in the input space." This is the crux selling point of the paper, and it is accomplished using integrated (and projected) jacobians. This technique also lets us see the contribution of a feature from a corpus example.  The authors then proceed to (briefly) characterize their model on MNIST against dKNN inspired methods, and show a use case on a clinical risk dataset.  They claim the following novelty:  1. Freedom to choose comparison corpus, no need for this to equal the training set 2. The instance-level decompositions are valid in latent and output space, and are more robust than other explainability methods

abstractText: Modern machine learning models are complicated. Most of them rely on convoluted latent representations of their input to issue a prediction. To achieve greater transparency than a black-box that connects inputs to predictions, it is necessary to gain a deeper understanding of these latent representations. To that aim, we propose SimplEx: a user-centred method that provides example-based explanations with reference to a freely selected set of examples, called the corpus. SimplEx uses the corpus to improve the user’s understanding of the latent space with post-hoc explanations answering two questions: (1) Which corpus examples explain the prediction issued for a given test example? (2) What features of these corpus examples are relevant for the model to relate them to the test example? SimplEx provides an answer by reconstructing the test latent representation as a mixture of corpus latent representations. Further, we propose a novel approach, the Integrated Jacobian, that allows SimplEx to make explicit the contribution of each corpus feature in the mixture. Through experiments on tasks ranging from mortality prediction to image classification, we demonstrate that these decompositions are robust and accurate. With illustrative use cases in medicine, we show that SimplEx empowers the user by highlighting relevant patterns in the corpus that explain model representations. Moreover, we demonstrate how the freedom in choosing the corpus allows the user to have personalized explanations in terms of examples that are meaningful for them.

title: The Mythos of Model Interpretability

venue: Communications of the ACM,

authors:
Travers Ching
Daniel S. Himmelstein
Brett K. Beaulieu-Jones
Alexandr A. Kalinin
Brian T. Do
Gregory P. Way
Enrico Ferrero
Paul-Michael Agapow
Michael Zietz
Michael M. Hoffman
Wei Xie
Gail L. Rosen
Benjamin J. Lengerich
Johnny Israeli
Jack Lanchantin
Stephen Woloszynek
Anne E. Carpenter
Avanti Shrikumar
Jinbo Xu
Evan M. Cofer
Christopher A. Lavender
Srinivas C. Turaga
Amr M. Alexandari
Zhiyong Lu
David J. Harris
Dave DeCaprio
Yanjun Qi
Anshul Kundaje
Yifan Peng
Laura K. Wiley
Marwin H.S. Segler
Simina M. Boca
S. Joshua Swamidass
Austin Huang
Anthony Gitter
Casey S. Greene

title: Opportunities and obstacles for deep learning in biology and medicine

venue: Journal of The Royal Society Interface,

authors:
Erico Tjoa
Cuntai Guan

title: A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI

venue: IEEE Transactions on Neural Networks and Learning Systems,

authors:
Alejandro Barredo Arrieta
Natalia Díaz-Rodríguez
Javier Del Ser
Adrien Bennetot
Siham Tabik
Alberto Barbado
Salvador Garcia
Sergio Gil-Lopez
Daniel Molina
Richard Benjamins
Raja Chatila
Francisco Herrera

title: Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI

venue: Information Fusion,

authors:
Arun Das
Paul Rad

title: Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey

title: Explainable AI: from black box to glass box

venue: Journal of the Academy of Marketing Science

title: A value for n-person games

venue: Contributions to the Theory of Games,

authors:
Anupam Datta
Shayak Sen
Yair Zick

title: Algorithmic Transparency via Quantitative Input Influence: Theory and Experiments with Learning Systems

venue: In Proceedings - 2016 IEEE Symposium on Security and Privacy,

authors:
Scott Lundberg
Su-In Lee

title: A Unified Approach to Interpreting Model Predictions

venue: Advances in Neural Information Processing Systems,

authors:
Marco Tulio Ribeiro
Sameer Singh
Carlos Guestrin

title: Why should i trust you?" Explaining the predictions of any classifier

venue: In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,

authors:
Mukund Sundararajan
Ankur Taly
Qiqi Yan

title: Axiomatic Attribution for Deep Networks

venue: 34th International Conference on Machine Learning, ICML 2017,

authors:
Amit Dhurandhar
Pin-Yu Chen
Ronny Luss
Chun-Chen Tu
Paishun Ting
Karthikeyan Shanmugam
Payel Das

title: Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives

authors:
Ruth C. Fong
Andrea Vedaldi

title: Interpretable Explanations of Black Boxes by Meaningful Perturbation

venue: Proceedings of the IEEE International Conference on Computer Vision,

authors:
Ruth Fong
Mandela Patrick
Andrea Vedaldi

title: Understanding deep networks via extremal perturbations and smooth masks

authors:
Jonathan Crabbé
Mihaela van der Schaar

title: Explaining Time Series Predictions with Dynamic Masks

venue: International Conference on Machine Learning,

authors:
Giang Nguyen
Daeyoung Kim
Anh Nguyen

title: The effectiveness of feature attribution methods and its correlation with automatic evaluation scores

authors:
R. Caruana
H. Kangarloo
J.D. Dionisio
U. Sinha
D. Johnson

title: Case-based explanation of non-case-based learning methods

venue: AMIA Symposium,

authors:
Isabelle Bichindaritz
Cindy Marling

title: Case-based reasoning in the health sciences: What’s next

venue: In Artificial Intelligence in Medicine,

authors:
Mark T. Keane
Eoin M. Kenny

title: How Case-Based Reasoning Explains Neural Networks: A Theoretical Analysis of XAI Using Post-Hoc Explanation-by-Example from a Survey of ANN-CBR Twin-Systems

venue: In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics),

authors:
Been Kim
Cynthia Rudin
Julie Shah

title: The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classification

authors:
Been Kim
Rajiv Khanna
Oluwasanmi Koyejo

title: Examples are not Enough, Learn to Criticize! Criticism for Interpretability

authors:
Karthik S. Gurumoorthy
Amit Dhurandhar
Guillermo Cecchi
Charu Aggarwal

title: Efficient Data Representation by Selecting Prototypes with Importance Weights

venue: Proceedings - IEEE International Conference on Data Mining, ICDM,

authors:
R. Dennis Cook
Sanford Weisenberg

title: Residuals and influence in regression

authors:
Pang Wei Koh
Percy Liang

title: Understanding Black-box Predictions via Influence Functions

authors:
Amirata Ghorbani
James Zou

title: Data Shapley: Equitable Valuation of Data for Machine Learning

venue: 36th International Conference on Machine Learning,

authors:
Amirata Ghorbani
Michael P. Kim
James Zou

title: A Distributional Framework for Data Valuation

authors:
Been Kim
Martin Wattenberg
Justin Gilmer
Carrie Cai
James Wexler
Fernanda Viegas
Rory Sayres

title: Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)

venue: 35th International Conference on Machine Learning,

authors:
Nicolas Papernot
Patrick McDaniel

title: Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning

authors:
Chih-Kuan Yeh
Joon Sik Kim
Ian E.H. Yen
Pradeep Ravikumar

title: Representer Point Selection for Explaining Deep Neural Networks

title: The Twin-System Approach as One Generic Solution for XAI: An Overview of ANN-CBR Twins for Explaining Deep Learning

title: The mnist database of handwritten digit images for machine learning research

venue: IEEE Signal Processing Magazine,

authors:
Eric Wu
Kevin Wu
Roxana Daneshjou
David Ouyang
Daniel E Ho
James Zou

title: How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals

venue: Nature Medicine,

authors:
Pan Ji
Tong Zhang
Hongdong Li
Mathieu Salzmann
Ian Reid

title: Deep Subspace Clustering Networks

authors:
Rushil Anirudh
Jayaraman J. Thiagarajan
Rahul Sridhar
Peer-Timo Bremer

title: MARGIN: Uncovering Deep Neural Networks using Graph

venue: Signal Analysis. arXiv,

authors:
Gregory Cohen
Saeed Afshar
Jonathan Tapson
André van Schaik

title: EMNIST: an extension of MNIST to handwritten letters

title: Using Synthetic Controls: Feasibility, Data Requirements, and Methodological Aspects

venue: Journal of Economic Literature,

authors:
Alberto Abadie
Alexis Diamond
Jens Hainmueller

title: Synthetic control methods for comparative case studies: Estimating the effect of California’s Tobacco control program

venue: Journal of the American Statistical Association,

authors:
Susan Athey
Mohsen Bayati
Nikolay Doudchenko
Guido Imbens
Khashayar Khosravi

title: Matrix Completion Methods for Causal Panel Data Models

authors:
Muhammad Amjad
Devavrat Shah
Dennis Shen

title: Robust Synthetic Control

venue: Journal of Machine Learning Research,

authors:
Jennifer Gordetsky
Jonathan Epstein

title: Grading of prostatic adenocarcinoma: current state and prognostic implications

venue: Diagnostic Pathology,

heading: 1 Introduction and related work

text:
How can we make a machine learning model convincing? If accuracy is undoubtedly necessary, it is rarely sufficient. As these models are used in critical areas such as medicine, finance and the criminal justice system, their black-box nature appears as a major issue [1, 2, 3]. With the necessity to address this problem, the landscape of explainable artificial intelligence (XAI) developed [4, 5]. A first approach in XAI is to focus on white-box models that are interpretable by design. However, restricting to a class of inherently interpretable models often comes at the cost of lower prediction accuracy [6]. In this work, we rather focus on post-hoc explainability techniques. These methods aim at improving the interpretability of black-box models by complementing their predictions with various kinds of explanations. In this way, it is possible to understand the prediction of a model without sacrificing its prediction accuracy.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Feature importance explanations are undoubtedly the most widespread type of post-hoc explanations. Popular feature importance methods include SHAP [7, 8, 9], LIME [10], Integrated Gradients [11], Contrastive Examples [12] and Masks [13, 14, 15]. These methods complement the model prediction for an input example with a score attributed to each input feature. This score reflects the importance of each feature for the model to issue its prediction. Knowing which features are important for a model prediction certainly provides more information on the model than the prediction by itself. However, these methods do not provide a reason as to why the model pays attention to these particular features.
Another approach is to contextualize each model prediction with the help of relevant examples. In fact, recent works [16] have demonstrated that human subjects often find example-based explanations more insightful than feature importance explanations. Complementing the model’s predictions with relevant examples previously seen by the model is commonly known as Case-Based Reasoning (CBR) [17, 18, 19]. The implementations of CBR generally involve models that create a synthetic representation of the dataset, where examples with similar patterns are summarized by prototypes [20, 21, 22]. At inference time, these models relate new examples to one or several prototypes to issue a prediction. In this way, the patterns that are used by the model to issue a prediction are made explicit with the help of relevant prototypes. A limitation of this approach is the restricted model architecture. The aforementioned procedure requires to opt for a family of models that rely on prototypes to issue a prediction. This family of model might not always be the most suitable for the task at hand. This motivates the development of generic post-hoc methods that make few or no assumption on the model.
The most common approach to provide example-based explanations for a wide variety of models mirrors feature importance methods. The idea is to complement the model prediction by attributing a score to each training example. This score reflects the importance of each training example for the model to issue its prediction. This score will typically be computed by simulating the effect of removing each training instance from the training set on the learned model [23]. Popular examples of such methods include Influence Functions [24] and Data-Shapley [25, 26]. These methods offer the advantage of being flexible enough to be used with a wide variety of models. They produce scores that describe what the model could have predicted if some examples were absent from the training set. This is very interesting in a data valuation perspective. However, in an explanation perspective, it is not clear how to reconstruct the model predictions with these importance scores.
So far, we have only discussed works that provide explanations of a model output, which is the tip of the iceberg. Modern machine learning models involve many convoluted transformations to deduce the output from an input. These transformations are expressed in terms of intermediate variables that are often called latent variables. Some treatment of these latent variables is necessary if we want to provide explanations that take the model complexity into account. This motivates several works that push the explainability task beyond the realm of model outputs. Among the most noticeable contributions in this endeavour, we cite Concept Activation Vectors that create a dictionary between human friendly concepts (such as the presence of stripes in an image) and their representation in terms of latent vectors [27]. Another interesting contribution is the Deep k-Nearest Neighbors model that contextualizes the prediction for an example with its Nearest Neighbours in the space of latent variables, the latent space [28]. An alternative exploration of the latent space is offered by the representer theorem that allows, under restrictive assumptions, to use latent vectors to decompose a model’s prediction in terms of its training examples [29].
Contribution In this work, we introduce a novel approach called SimplEx that lies at the crossroad of the above research directions. SimplEx outputs post-hoc explanations in the form of Figure 1, where the model’s prediction and latent representation for a test example is approximated as a mixture of examples extracted from a corpus of examples.
In each case, SimplEx highlights the role played by each feature of each corpus example in the latent space decomposition. SimplEx centralizes many functionalities that, to the best of our knowledge, constitute a leap forward from the previous state of the art. (1) SimplEx gives the user freedom to choose the corpus of examples whom with the model’s predictions are decomposed. Unlike previous methods such as the representer theorem, there is no need for this corpus of examples to be equal to the model’s training set. This is particularly interesting for two reasons: (a) the training set of a model is not always accessible (b) the user might want explanations in terms of examples that make sense for them. For instance, a doctor might want to understand the predictions of a risk model in terms of patients they know. (2) The decompositions of SimplEx are valid, both in latent and output space. We show that, in both cases, the corpus mixtures discovered by SimplEx offer significantly more precision and robustness than previous methods such as Deep k-Nearest Neighbors and the representer theorem. (3) SimplEx details the role played by each feature in the corpus mixture. This is done by introducing Integrated Jacobians, a generalization of Integrated Gradients that makes the contribution of each corpus feature explicit in the latent space decomposition. This creates a bridge between two research directions that have mostly developed independently: feature importance and example-based explanations [19, 30]. In Section 3 of the supplementary material, we report a user-study involving 10 clinicians. This study supports the significance of our contribution.

heading: 2 SimplEx

text: In this section, we formulate our method rigorously. Our purpose is to explain the black-box prediction for an unseen test example with the help of a set of known examples that we call the corpus. We start with a clear statement of the family of black-boxes for which our method applies. Then, we detail how the set of corpus examples can be used to decompose a black-box representation for the unseen example. Finally, we show that the corpus decomposition can offer explanations at the feature level.

heading: 2.1 Preliminaries

text:
Let X ⊆ RdX be an input (or feature) space and Y ⊆ RdY be an output (or label) space, where dX and dY are respectively the dimension of the input and the output space. Our task is to explain individual predictions of a given black-box f : X → Y . In order to build our explainability method, we need to make an assumption on the family of black-boxes that we wish to interpret. Assumption 2.1 (Black-box Restriction). We restrict to black-boxes f : X → Y that can be decomposed as f = l ◦ g, where g : X → H maps an input x ∈ X to a latent vector h = g (x) ∈ H and l : H → Y linearly maps1 a latent vector h ∈ H to an output y = l(h) = Ah ∈ Y . In the following, we call H ⊆ RdH the latent space. Typically, this space has higher dimension than the output space dH > dY . Remark 2.1. In the context of deep-learning, this assumption requires that the last hidden layer maps linearly to the output. While it is often the case, it is crucial in the following since we will use the fact that linear combinations in latent space correspond to linear combinations in output space. Our purpose is to gain insights on the structure of the latent space. Remark 2.2. This assumption is compatible with regression and classification models, we just need to clarify what we mean by output in the case of classification. If f is a classification black-box that predicts the probabilities for each class, it will typically take the form in Assumption 2.1 up to a normalizing map φ (typically a softmax): f = φ ◦ l ◦ g. In this case, we ignore2 the normalizing map φ and define the output to be y = (l ◦ g) (x). Our explanations for f rely on a set of examples that we call the corpus. These examples will typically (but not necessarily) be a representative subset of the black-box training set. The corpus set has to be understood as a set of reference examples that we want to use as building blocks to interpret unseen examples. In order to index these examples, it will be useful to denote by [n1 : n2] the set of natural numbers between the natural numbers n1 and n2 with n1 < n2. Further, we denote [n] = [1 : n] the set of natural numbers between 1 and n ≥ 1. The corpus of examples is a set C = {xc | c ∈ [C]} containing C ∈ N∗ examples xc ∈ X . In the following, superscripts are labels for examples and subscripts are labels for vector components. In this way, xci has to be understood as the component i of corpus example c.
1The map can in fact be affine. In the following, we omit the bias term b ∈ Y that can be reabsorbed in g. 2There is no loss of information as the output allows us to reconstruct class probabilities p via p = φ (y).

heading: 2.2 A corpus of examples to explain a latent representation

text:
Our purpose is to understand a prediction f(x) for an unseen test example x with the help of the corpus. How can we decompose the prediction f(x) in terms of corpus predictions f(xc)? A naive attempt would be to express x as a mixture of inputs from the corpus C: x = ∑Cc=1 wcxc with weights wc ∈ [0, 1] that sum to one ∑Cc=1 wc = 1. The weakness of this approach is that the signification of the mixture weights is not conserved if the black-box f is not a linear map: f( ∑C c=1 w cxc) 6=∑Cc=1 wcf(xc).
Fortunately, Assumption 2.1 offers us a better vector space to perform a corpus decomposition of the unseen example x. We first note that the map g induces a latent representation of the corpus g(C) = {hc = g(xc) | xc ∈ C} ⊂ H. Similarly, x has a latent representation h = g(x) ∈ H. Following the above line of reasoning, we could therefore perform a corpus decomposition in latent space h = ∑C c=1 w
chc. Now, by using the linearity of l, we can compute the black-box output of this mixture in latent space: l( ∑C c=1 w chc) = ∑C c=1 w cl(hc). In this case, the weights that are used to decompose the latent representation h in terms of the latent representation of the corpus g(C) also reflect the way in which the black-box prediction f(x) can be decomposed in terms of the corpus outputs f(C). This hints that the latent spaceH is endowed with the appropriate geometry to make corpus decompositions. More formally, we think in terms of the convex hull spanned by the corpus. Definition 2.1 (Corpus Hull). The corpus convex hull spanned by a corpus C with latent representation g(C) = {hc = g(xc) | xc ∈ C} ⊂ H is the convex set
CH (C) = {
C∑ c=1 wchc ∣∣∣∣∣ wc ∈ [0, 1] ∀c ∈ [C] ∧ C∑ c=1 wc = 1 } .
Remark 2.3. This is the set of latent vectors that are a mixture of the corpus latent vectors.
At this stage, it is important to notice that an exact corpus decomposition is not possible if h 6∈ CH(C). In such a case, the best we can do is to find the element ĥ ∈ CH(C) that best approximates h. IfH is endowed with a norm ‖ · ‖H, this corresponds to the convex optimization problem
ĥ = argmin h̃ ∈ CH(C) ‖ h− h̃ ‖H . (1)
By definition, the corpus representation ĥ of h can be expanded3 as a mixture of elements from g(C): ĥ =∑Cc=1 wchc. The weight can naturally be interpreted as a measure of importance in the reconstruction of h with the corpus. Clearly, wc ≈ 0 for some c ∈ [C] indicates that hc does not play a significant role in the corpus representation ĥ of h. On the other hand, wc ≈ 1 indicates that hc generates the corpus representation ĥ by itself.
At this stage, a natural question arises: how can we know if the corpus approximation ĥ is a good approximation for h? The answer is given by the residual vector h− ĥ that measures the shift between the latent representation h = g(x) and the corpus hull CH(C). It is natural to use this residual vector to detect examples that cannot be explained with the selected corpus of examples C. Definition 2.2 (Corpus Residual). The corpus residual associated to a latent vector h ∈ H and its corpus representation ĥ ∈ CH(C) solving (1) is the quantity
rC(h) =‖ h− ĥ ‖H= min h̃ ∈ CH(C) ‖ h− h̃ ‖H .
gx1
x2 x3
h1
h2
h3x h
ĥ
Figure 2: Corpus convex hull and residual.
In Section 1.1 of the supplementary material, we show that the corpus residual also controls the quality of the corpus approximation in output space Y . All the corpus-related quantities that we have introduced so far are summarized visually in Figure 2. Note that this Figure is a simplification of the reality as C will typically
be larger than 3 and dX , dH will typically be higher than 2. We are now endowed with a rigorous way to decompose a test example in terms of corpus examples in latent space. In the next section, we detail how to pull-back this decomposition to input space.
3Note that this decomposition might not be unique, more details in Section 1 of the supplementary material.

heading: 2.3 Transferring the corpus explanation in input space

text:
Now that we are endowed with a corpus decomposition ĥ = ∑C
c=1 w chc that approximates h, it
would be convenient to have an understanding of the corpus decomposition in input space X . For the sake of notation, we will assume that the corpus approximation is good so that it is unnecessary to draw a distinction between the latent representation h of the unseen example x and its corpus decomposition ĥ. If we want to understand the corpus decomposition in input space, a natural approach [11] is to fix a baseline input x0 together with its latent representation h0 = g(x0). Let us now decompose the representation shift h− h0 in terms of the corpus:
h− h0 = C∑
c=1
wc ( hc − h0 ) . (2)
With this decomposition, we understand the total shift in latent space h− h0 in terms of individual contributions from each corpus member. In the following, we focus on the comparison between the baseline and a single corpus example xc together with its latent representation hc by keeping in mind that the full decomposition (2) can be reconstructed with the whole corpus. To bring the discussion in input spaceX , we interpret the shift in latent space hc−h0 as resulting from a shift xc−x0 in the input space. We are interested in the contribution of each feature to the latent space shift. To decompose the shift in latent space in terms of the features, we parametrize the shift in input space with a line γc : [0, 1] → X that goes from the baseline to the corpus example: γc(t) = x0 + t · (xc − x0) for t ∈ [0, 1]. Together with the black-box, this line induces a curve in latent space g ◦ γc : [0, 1]→ H that goes from the baseline latent representation h0 to the corpus example latent representation hc. Let us now use an infinitesimal decomposition of this curve to make the contribution of each input feature explicit. If we assume that g is differentiable at γc(t), we can use a first order approximation of the curve at the vicinity of t ∈ (0, 1) to decompose the infinitesimal shift in latent space:
g ◦ γc(t+ δt)− g ◦ γc(t)︸ ︷︷ ︸ Infinitesimal shift in latent space = dX∑ i=1 ∂g ∂xi ∣∣∣∣ γc(t) dγci dt ∣∣∣∣ t δt+ o(δt)
= dX∑ i=1 ∂g ∂xi ∣∣∣∣ γc(t) (xci − x0i ) · δt+ o(δt),
where we used γci (t) = x 0 i + t · (xci − x0i ) to obtain the second equality. In this decomposition, each input feature contributes additively to the infinitesimal shift in latent space. It follows trivially that the contribution of the input feature corresponding to input dimension i ∈ [dX ] is given by
δjci (t) = (x c i − x0i ) · ∂g ∂xi ∣∣∣∣ γc(t) δt ∈ H.
In order to compute the overall contribution of feature i to the shift, we let δt→ 0 and we sum the infinitesimal contributions along the line γc. If we assume4 that g is almost everywhere differentiable, this sum converges to an integral in the limit δt→ 0 . This motivates the following definitions. Definition 2.3 (Integrated Jacobian & Projection). The integrated Jacobian between a baseline (x0,h0 = g(x0)) and a corpus example (xc,hc = g(xc)) ∈ X ×H associated to feature i ∈ [dX ] is
jci = ( xci − x0i ) ∫ 1 0 ∂g ∂xi ∣∣∣∣ γc(t) dt ∈ H,
where γc(t) ≡ x0 + t · ( xc − x0 ) for t ∈ [0, 1]. This vector indicates the shift in latent space induced by feature i of corpus example c when comparing the corpus example with the baseline. To summarize this contribution to the shift h− h0 described in (2), we define the projected Jacobian
pci = projh−h0 (j c i ) ≡ 〈 h− h0 , jci 〉 〈 h− h0 , h− h0 〉
∈ R,
where 〈·,·〉 is an inner product forH and the normalization is chosen for the purpose of Proposition 2.1. Remark 2.4. The integrated Jacobian can be seen as a latent-space generalization of Integrated Gradients [11]. In Section 1.3 of the supplementary material, we establish the relationship between the two quantities: IGci = l(j c i ).
4This is not restrictive, DNNs with ReLU activation functions satisfy this assumption for instance.
We summarize the Jacobian quantities in Figure 3. By inspecting the figure, we notice that projected Jacobians encode the contribution of feature i from corpus example c to the overall shift in latent space: pci > 0 implies that this feature creates a shift pointing in the same direction as the overall shift; pci < 0 implies that this feature creates a shift pointing in the opposite direction and pci = 0 implies that this feature creates a shift in an orthogonal direction. We
use the projections to summarize the contribution of each feature in Figures 1 , 8 & 9. The colors green and red indicate respectively a positive and negative projection. In addition to these geometrical insights, Jacobian quantities come with natural properties.
Proposition 2.1 (Properties of Integrated Jacobians). Consider a baseline (x0,h0 = g(x0)) and a test example together with their latent representation (x,h = g(x)) ∈ X × H. If the shift h − h0 admits a decomposition (2), the following properties hold.
(A) : C∑
c=1 dX∑ i=1 wcjci = h− h0 (B) : C∑ c=1 dX∑ i=1 wcpci = 1.
Proof. The proof is provided in Section 1.4 of the supplementary material.
These properties show that the integrated Jacobians and their projections are the quantities that we are looking for: they transfer the corpus explanation into input space. The first equality decomposes the shift in latent space in terms of contributions wcjci arising from each feature of each corpus example. The second equality sets a natural scale to the contribution of each feature. For this reason, it is natural to use wcpci to measure the contribution of feature i of corpus example c.

heading: 3 Experiments

text: In this section, we evaluate quantitatively several aspects of our method. In a first experiment, we verify that the corpus decomposition scheme described in Section 2 yields good approximations for the latent representation of test examples extracted from the same dataset as the corpus examples. In a realistic clinical use case, we illustrate the usage of SimplEx in a set-up where different corpora reflecting different datasets are used. The experiments are summarized below. In Section 2 of the supplementary material, we provide more details and further experiments with time series and synthetic data. The code for our method and experiments is available on the Github repository https://github.com/JonathanCrabbe/Simplex. All the experiments have been replicated on different machines.

heading: 3.1 Precision of corpus decomposition

text:
Description The purpose of this experiment is to check if the corpus decompositions described in Section 2 allows us to build good approximations of the latent representation of test examples. We start with a datasetD that we split into a training setDtrain and a testing setDtest. We train a black-box f for a given task on the training set Dtrain. We randomly sample a set of corpus examples from the training set C ⊂ Dtrain (we omit the true labels for the corpus examples) and a set of test examples from the testing set T ⊂ Dtest. For each test example x ∈ T , we build an approximation ĥ for h = g(x) with the corpus examples latent representations. In each case, we let the method use only K corpus examples to build the approximation. We repeat the experiment for several values of K.
Metrics We are interested in measuring the precision of the corpus approximation in latent space and in output space. To that aim, we use the R2 score in both spaces. In this way, R2H measures the precision of the corpus approximation ĥ with respect to the true latent representation h. Similarly, R2Y measures the precision of the corpus approximation ŷ = l(ĥ) with respect to the true output y = l(h). Both of these metrics satisfy −∞ < R2 ≤ 1. A higher R2 score is better with R2 = 1
corresponding to a perfect approximation. All the metrics are computed over the test examples T . The experiments are repeated 10 times to report standard deviations across different runs.
Baselines We compare our method5 (SimplEx) with 3 baselines. A first approach, inspired by [28], consists in using the K-nearest corpus neighbours in latent space to build the latent approximation ĥ. Building on this idea, we introduce two baselines (1) KNN Uniform that takes the average latent representation of the K-nearest corpus neighbours of h in latent space (2) KNN Distance that computes the same average with weights wc inversely proportional to the distance ‖ h − hc ‖H. Finally, we use the representer theorem [29] to produce an approximation ŷ of y with the corpus C. Unlike the other methods, the representer theorem does not allow to produce an approximation in latent space.
Datasets We use two different datasets with distinct tasks for our experiment: (1) 240,486 patients enrolled in the American SEER program [31]. We consider the binary classification task of predicting cancer mortality for patients with prostate cancer. We train a multilayer perceptron (MLP) for this task. Since this task is simple, we show that a corpus of C = 100 patients yields good approximations. (2) 70,000 MNIST images of handwritten digits [32]. We consider the multiclass classification task of identifying the digit represented on each image. We train a convolutional neural network (CNN) for the image classification. This classification task is more complex than the previous one (higher dX and dY ), we show that a corpus of C = 1, 000 images yields good approximations in this case.
Results The results for SimplEx and the KNN baselines are presented in Figure 4 & 5. Several things can be deduced from these results: (1) It is generally harder to produce a good approximation in latent space than in output space as R2H < R 2 Y for most examples (2) SimplEx produces the most accurate approximations, both in latent and output space. These approximations are of high quality with R2 ≈ 1. (3) The trends are qualitatively different between SimplEx and the other baselines. The accuracy of SimplEx increases with K and stabilizes when a small number of corpus members contribute (K = 5 in both cases). The accuracy of the KNN baselines increases with K,
5To enforce SimplEx to select K examples, we add a L1 penalty making the C −K smallest weights vanish.
reaches a maximum for a small K and steadily decreases for larger K. This can be understood easily: whenK increases beyond the number of relevant corpus examples, irrelevant examples will be added in the decomposition. SimplEx will typically annihilate the effect of these irrelevant examples by setting their weights wc to zero in the corpus decomposition. The KNN baselines include the irrelevant corpus members in the decomposition, which alters the quality of the approximation. This suggests that K has to be tuned for each example with KNN baselines, while the optimal number of corpus examples to contribute is learned by SimplEx. (4) The standard deviations indicate that the performances of SimplEx are more consistent across different runs. This is particularly true in the prostate cancer experiment, where the corpus size C is smaller. This suggests that SimplEx is more robust than the baselines. (5) For the representer theorem, we have R2Y = −(6.6± 6.1) · 107 for the prostate cancer dataset and R2Y = −(7.2± 6.6) for MNIST. This corresponds to poor estimations of the black-box output. We propose some hypotheses to explain this observation in Section 2.1 of the supplementary material.

heading: 3.2 Significance of Jacobian Projections

text:
Description The purpose of this experiment is to check if SimplEx’s Jacobian Projections are a good measure of the importance for each corpus feature in constructing the test latent representation h. In the same setting as in the previous experiment, we start with a corpus C of C = 500 MNIST images. We build a corpus approximation for an example x ∈ X with latent representation h = g(x) ∈ H. The precision of this approximation is reflected by its corpus residual rC(h). For each corpus example xc ∈ C, we would like to identify the features that are the most important in constructing the corpus decomposition of h. With SimplEx, this is reflected by the Jacobian Projections pci . We evaluate these scores for each feature i ∈ [dX ] of each corpus example c ∈ [C]. For each corpus image xc ∈ C, we select the n most important pixels according to the Jacobian Projections and the baseline. In each case, we build a mask mc that replaces these n most important pixels by black pixels. This yields a corrupted corpus image xccor = mc xc, where denotes the Hadamard product. By corrupting all the corpus images, we obtain a corrupted corpus Ccor . We analyse how well this corrupted corpus approximates h, this yields a residual rCcor (h).
Metric We are interested in measuring the effectiveness of the corpus corruption. This is reflected by the metric δcor(h) = rCcor (h)− rC(h) . A higher value for this metric indicates that the features selected by the saliency method are more important for the corpus to produce a good approximation of h in latent space. We repeat this experiment for 100 test examples and for different numbers n of perturbed pixels.
Baseline As a baseline for our experiment, we use Integrated Gradients, which is close in spirit to our method. In a similar fashion, we compute the Integrated Gradients IGci for each feature i ∈ [dX ] of each corpus example c ∈ [C] and construct a corrupted corpus based on these scores. Results The results are presented in the form of box plots in Figure 6. We observe that the corruptions induced by the Jacobian Projections are significantly more impactful when few pixels are perturbed. The two methods become equivalent when more pixels are perturbed. This demonstrates that Jacobian Projections are more suitable to measure the importance of features when performing a latent space reconstruction, as it is the case for SimplEx.

heading: 3.3 Use case: clinical risk model across countries

text:
Very often, clinical risk models are produced and validated with the data of patients treated at a single site [33]. This can cause problems when these models are deployed at different sites for two reasons: (1) Patients from different sites can have different characteristics (2) Rules that are learned for one site might not be true for another site. One possible way to alleviate this problem would be to detect patients for which the model prediction is highly extrapolated and/or ambiguous. In this way, doctors from different sites can make an enlightened use of the risk model rather than blindly believing the model’s predictions. We demonstrate that SimplEx provides a natural framework for this set-up.
As in the previous experiment, we consider a dataset DUSA containing patients enrolled in the American SEER program [31]. We train and validate an MLP risk model with DUSA. To give a realistic realization of the above use-case, we assume that we want to deploy this risk model in a different site: the United Kingdom. For this purpose, we extract DUK from the set of 10,086 patients enrolled in the British Prostate Cancer UK program [34]. These patients are characterized by the
same features for both DUK and DUSA. However, the datasets DUK and DUSA differ by a covariate shift: patients from DUK are in general older and at earlier clinical stages. When comparing the two populations in terms of the model, a first interesting question to ask is whether the covariate shift between DUSA and DUK affects the model representation. To explore this question, we take a first corpus of American patients CUSA ⊂ DUSA. If there is indeed a difference in terms of the latent representations, we expect the representations of test examples fromDUK to be less closely approximated by their decomposition with respect to CUSA. If this is true, the corpus residuals associated to examples of DUK will typically larger than the ones associated to DUSA. To evaluate this quantitatively, we consider a mixed set of test examples T sampled from both DUK and DUSA: T ⊂ DUK t DUSA. We sample 100 examples from both sources: | T ∩ DUK| = |T ∩ DUSA| = 100. We then approximate the latent representation of each example h ∈ g(T ) and compute the associated corpus residual rCUSA(h). We sort the test examples from T by decreasing order of corpus residual and we use this sorted list to see if we can detect the examples from DUK. We use previous baselines for comparison, results are shown in Figure 7.
Several things can be deduced from this experiment. (1) The results strongly suggest that the difference between the two datasets DUSA and DUK is reflected in their latent representations. (2) The corpus residuals from SimplEx offer the most reliable way to detect examples that are different from the corpus examples CUSA. None of the methods matches the maximal baseline since some examples of DUSA resemble examples from DUK. (3) When the corpus examples are representative of the training set, as it is the case in the experiment, our approach based on SimplEx provides a systematic way to detect test examples that have representations that are different from the ones produced at training time. A doctor should be more sceptical with respect to model predictions associated
to larger residual with respect to CUSA as these arise from an extrapolation region of the latent space. Let us now make the case more concrete. Suppose that an American and a British doctor use the above risk model to predict the outcome for their patients. Each doctor wants to decompose the predictions of the model in terms of patients they know. Hence, the American doctor selects a corpus of American patients CUSA ⊂ DUSA and the British doctor selects a corpus of British patients CUK ⊂ DUK. Both corpora have the same size CUSA = CUK = 1, 000. We suppose that the doctors know the model prediction and the true outcome for each patient in their corpus. Both doctors are sceptical about the risk model and want to use SimplEx to decide when it can be trusted. This leads them to a natural question: is it possible to anticipate misclassification with the help of SimplEx?
In Figure 8 & 9, we provide two typical examples of misclassified British patients from DUK \ CUK together with their decomposition in terms of the two corpora CUSA and CUK. These two examples
exhibit two qualitatively different situations. In Figure 8, both the American and the British doctors make the same observation: the model relates the test patient to corpus patients that are mostly misclassified by the model. With the help of SimplEx, both doctors will rightfully be sceptical with respect to the model’s prediction.
In Figure 9, something even more interesting occurs: the two corpus decompositions suggest different conclusions. In the American doctor’s perspective, the prediction for this patient appears perfectly coherent as all patients in the corpus decomposition have very similar features and all of them are rightfully classified. On the other hand, the British doctor will reach the opposite conclusion as the most relevant corpus patient is misclassified by the model. In this case, we have a perfect illustration of the limitation of the transfer of a risk model from one site (America) to another (United Kingdom): similar patients from different sites can have different outcomes. In both cases, since the test patient is British, only the decomposition in terms of CUK really matters. In both cases, the British doctor could have anticipated the misclassification of each patient with SimplEx.

heading: 4 Discussion

text:
We have introduced SimplEx, a method that decomposes the model representations at inference time in terms of a corpus. Through several experiments, we have demonstrated that these decompositions are accurate and can easily be personalized to the user. Finally, by introducing Integrated Jacobians, we have brought these explanations to the feature level.
We believe that our bridge between feature and example-based explainability opens up many avenues for the future. A first interesting extension would be to investigate how SimplEx can be used to understand latent representations involved in unsupervised learning. For instance, SimplEx could be used to study the interpretability of self-expressive latent representations learned by autoencoders [35]. A second interesting possibility would be to design a rigorous scheme to select the optimal corpus for a given model and dataset. Finally, a formulation where we allow the corpus to vary on the basis of observations would be particularly interesting for online learning.

heading: Acknowledgement

text: The authors are grateful to Alicia Curth, Krzysztof Kacprzyk, Boris van Breugel, Yao Zhang and the 4 anonymous NeurIPS 2021 reviewers for their useful comments on an earlier version of the manuscript. Jonathan Crabbé would like to thank Bogdan Cebere for replicating the experiments in this manuscript. Jonathan Crabbé would like to acknowledge Bilyana Tomova for many insightful discussions and her constant support. Jonathan Crabbé is funded by Aviva. Fergus Imrie is supported by by the National Science Foundation (NSF), grant number 1722516. Zhaozhi Qian and Mihaela van der Schaar are supported by the Office of Naval Research (ONR), NSF 1722516.

title: Explaining Latent Representations with a Corpus of Examples


summaries: This paper introduces Swin Transformer in the visual domain for vision-language pretraining. Previous VL pretraining models feed image tokens and text tokens together into the cross-modal transformer, and the visual relationship learning and inter-modal alignment are encapsulated in the same transformer network. The authors propose that visual relationship between different visual objects or concepts is important for inter-modal alignment learning, and there should be independent processing of visual relationship in the framework. Motivated by this hypothesis, the authors add a visual-only transformer to learn visual relationship before feeding the image and text tokens together to the cross-modal transformer. In addition, the authors propose a metric named inter-modality flow (IMF) to measure the cross-modal interactions. They also propose masked feature regression as a new pretraining task. Pretraining is conducted on in-domain datasets COCO and Visual Genome. Evaluation is conducted on three downstream tasks VQA, visual reasoning, and visual entailment.

summaries: This paper proposed to use the vision transformer as the visual backbone for vision-language pre-training. To quantize the information flow transferred between vision and language modalities, the authors proposed a metric called Inter-modality flow that aggregates the attention weight across different layers. Experiments on VQA, visual entailment, and visual reasoning validates that the proposed method achieves state-of-the-art performance.

summaries: This paper provides a novel way of applying masked feature regression (MFR) to transformer-based visual encoders. In addition to this new way of MRM, the paper also provides a quantifiable measurement of how two modalities (image and text) interchange their information with the aids from the attention flow method. This quantity, the inter-modality flow (IMF), is then used to analyze the characteristics of different vision-and-language pretraining models.

summaries: This paper proposes a transformer based visual backbone for vision and language understanding. It uses a vision transformer backbone (Swin Transformer) to extract visual representation, which is then passed to a mask generation module. The visual tokens(including masked tokens) and text tokens embeddings are fed to a multimodal transformer model, where different pretraining objectives are applied. This architecture achieves state-of-the-art results in VQA, SNLI-VE, NLVR^2, Image-Text Retrieval. Authors propose an Inter Modality Flow metric that measure interaction between the visual and textual modalities and correlates well with the downstream results.

abstractText: Vision-Language Pre-training (VLP) aims to learn multi-modal representations from image-text pairs and serves for downstream vision-language tasks in a finetuning fashion. The dominant VLP models adopt a CNN-Transformer architecture, which embeds images with a CNN, and then aligns images and texts with a Transformer. Visual relationship between visual contents plays an important role in image understanding and is crucial for inter-modal alignment learning in VLP. However, CNNs have limitations in visual relation learning due to local receptive field’s weakness in modeling long-range dependencies. Thus the two objectives of learning visual relation and inter-modal alignment are encapsulated in the same Transformer network. Such design might restrict the inter-modal alignment learning in the Transformer by neglecting the specialized characteristic of each objective. To tackle this challenge, we propose a fully Transformer visual embedding for VLP to better learn visual relation and further promote inter-modal alignment. Specifically, we propose a metric named Inter-Modality Flow (IMF) to measure the interaction between vision and language (i.e., inter-modality). We also design a novel masking optimization mechanism named Masked Feature Regression (MFR) in Transformer to further promote the inter-modality learning. To the best of our knowledge, this is the first study to explore the benefit of Transformer for visual feature learning in VLP. We verify our method on a wide range of vision-language tasks, including Image-Text Retrieval, Visual Question Answering (VQA), Visual Entailment and Visual Reasoning. The result shows that our approach not only outperforms the state-of-the-art VLP models, but also exhibits superiority on the IMF metric.

authors:
Samira Abnar
Willem Zuidema

title: Quantifying attention flow in transformers

venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,

authors:
Peter Anderson
Xiaodong He
Chris Buehler
Damien Teney
Mark Johnson
Stephen Gould
Lei Zhang

title: Bottom-up and top-down attention for image captioning and visual question 10 answering

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Josh Beal
Eric Kim
Eric Tzeng
Dong Huk Park
Andrew Zhai
Dmitry Kislyuk

title: Toward transformer-based object detection

venue: arXiv preprint arXiv:2012.09958,

authors:
Hedi Ben-Younes
Rémi Cadene
Matthieu Cord
Nicolas Thome

title: Mutan: Multimodal tucker fusion for visual question answering

venue: In Proceedings of the IEEE international conference on computer vision,

authors:
Samuel R Bowman
Gabor Angeli
Christopher Potts
Christopher D Manning

title: A large annotated corpus for learning natural language inference

venue: In EMNLP,

authors:
Jize Cao
Zhe Gan
Yu Cheng
Licheng Yu
Yen-Chun Chen
Jingjing Liu

title: Behind the scene: Revealing the secrets of pre-trained vision-and-language models

venue: In European Conference on Computer Vision,

authors:
Nicolas Carion
Francisco Massa
Gabriel Synnaeve
Nicolas Usunier
Alexander Kirillov
Sergey Zagoruyko

title: End-to-end object detection with transformers

authors:
Yen-Chun Chen
Linjie Li
Licheng Yu
Ahmed El Kholy
Faisal Ahmed
Zhe Gan
Yu Cheng
Jingjing Liu

title: Uniter: Universal image-text representation learning

authors:
Jia Deng
Wei Dong
Richard Socher
Li-Jia Li
Kai Li
Li Fei-Fei

title: Imagenet: A largescale hierarchical image database

venue: IEEE conference on computer vision and pattern recognition,

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova. Bert

title: Pre-training of deep bidirectional transformers for language understanding

authors:
Virginie Do
Oana-Maria Camburu
Zeynep Akata
Thomas Lukasiewicz

title: e-snli-ve- 2.0: Corrected visual-textual entailment with natural language explanations

venue: arXiv preprint arXiv:2004.03744,

authors:
Alexey Dosovitskiy
Lucas Beyer
Alexander Kolesnikov
Dirk Weissenborn
Xiaohua Zhai
Thomas Unterthiner
Mostafa Dehghani
Matthias Minderer
Georg Heigold
Sylvain Gelly

title: An image is worth 16x16 words: Transformers for image recognition at scale

venue: arXiv preprint arXiv:2010.11929,

authors:
Fartash Faghri
David J Fleet
Jamie Ryan Kiros
Sanja Fidler

title: Vse++: Improving visualsemantic embeddings with hard negatives

venue: arXiv preprint arXiv:1707.05612,

authors:
Jianlong Fu
Tao Mei
Kuiyuan Yang
Hanqing Lu
Yong Rui

title: Tagging personal photos with transfer deep learning

venue: In Proceedings of the 24th International Conference on World Wide Web,

authors:
Jianlong Fu
Jinqiao Wang
Yong Rui
Xin-Jing Wang
Tao Mei
Hanqing Lu

title: Image tag refinement with view-dependent concept representations

venue: IEEE Transactions on Circuits and Systems for Video Technology,

authors:
Jun Fu
Jing Liu
Haijie Tian
Yong Li
Yongjun Bao
Zhiwei Fang
Hanqing Lu

title: Dual attention network for scene segmentation

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
Yash Goyal
Tejas Khot
Douglas Summers-Stay
Dhruv Batra
Devi Parikh

title: Making the v in vqa matter: Elevating the role of image understanding in visual question answering

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

authors:
Yupan Huang
Hongwei Xue
Bei Liu
Yutong Lu

title: Unifying multimodal transformer for bi-directional image and text generation

venue: In Proceedings of the 29th ACM International Conference on Multimedia,

authors:
Zhicheng Huang
Zhaoyang Zeng
Yupan Huang
Bei Liu
Dongmei Fu
Jianlong Fu

title: Seeing out of the box: End-to-end pre-training for vision-language representation learning

venue: arXiv preprint arXiv:2104.03135,

authors:
Zhicheng Huang
Zhaoyang Zeng
Bei Liu
Dongmei Fu
Jianlong Fu

title: Pixel-bert: Aligning image pixels with text by deep multi-modal transformers

venue: arXiv preprint arXiv:2004.00849,

authors:
Drew A Hudson
Christopher D Manning

title: Compositional attention networks for machine reasoning

venue: In International Conference on Learning Representations,

authors:
Ganesh Jawahar
Benoît Sagot
Djamé Seddah

title: What does bert learn about the structure of language? In ACL 2019-57th Annual Meeting of the Association for Computational Linguistics, 2019

authors:
Huaizu Jiang
Ishan Misra
Marcus Rohrbach
Erik Learned-Miller
Xinlei Chen

title: In defense of grid features for visual question answering

authors:
Andrej Karpathy
Li Fei-Fei

title: Deep visual-semantic alignments for generating image descriptions

authors:
Jin-Hwa Kim
Jaehyun Jun
Byoung-Tak Zhang

title: Bilinear attention networks

venue: In Proceedings of the 32nd International Conference on Neural Information Processing Systems,

authors:
Wonjae Kim
Bokyung Son
Ildoo Kim

title: Vilt: Vision-and-language transformer without convolution or region supervision

venue: arXiv preprint arXiv:2102.03334,

authors:
Ranjay Krishna
Yuke Zhu
Oliver Groth
Justin Johnson
Kenji Hata
Joshua Kravitz
Stephanie Chen
Yannis Kalantidis
Li-Jia Li
David A Shamma

title: Visual genome: Connecting language and vision using crowdsourced dense image annotations

venue: International journal of computer vision,

authors:
Kuang-Huei Lee
Xi Chen
Gang Hua
Houdong Hu
Xiaodong He

title: Stacked cross attention for image-text matching

venue: In Proceedings of the European Conference on Computer Vision (ECCV),

authors:
Gen Li
Nan Duan
Yuejian Fang
Ming Gong
Daxin Jiang

title: Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training

venue: In Proceedings of the AAAI Conference on Artificial Intelligence,

authors:
Liunian Harold Li
Mark Yatskar
Da Yin
Cho-Jui Hsieh
Kai-Wei Chang

title: Visualbert: A simple and performant baseline for vision and language

authors:
Tsung-Yi Lin
Michael Maire
Serge Belongie
James Hays
Pietro Perona
Deva Ramanan
Piotr Dollár
C Lawrence Zitnick

title: Microsoft coco: Common objects in context

venue: In European conference on computer vision,

authors:
Bei Liu
Zhicheng Huang
Zhaoyang Zeng
Zheyu Chen
Jianlong Fu

title: Learning rich image region representation for visual question answering

authors:
Ze Liu
Yutong Lin
Yue Cao
Han Hu
Yixuan Wei
Zheng Zhang
Stephen Lin
Baining Guo

title: Swin transformer: Hierarchical vision transformer using shifted windows

venue: arXiv preprint arXiv:2103.14030,

authors:
Jiasen Lu
Dhruv Batra
Devi Parikh
Stefan Lee

title: Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks

authors:
Niki Parmar
Ashish Vaswani
Jakob Uszkoreit
Lukasz Kaiser
Noam Shazeer
Alexander Ku
Dustin Tran

title: Image transformer

venue: In International Conference on Machine Learning,

authors:
Bryan A Plummer
Liwei Wang
Chris M Cervantes
Juan C Caicedo
Julia Hockenmaier
Svetlana Lazebnik

title: Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models

authors:
Shaoqing Ren
Kaiming He
Ross B Girshick
Jian Sun

title: Faster r-cnn: Towards real-time object detection with region proposal networks

venue: In NIPS,

authors:
Weijie Su
Xizhou Zhu
Yue Cao
Bin Li
Lewei Lu
Furu Wei
Jifeng Dai

title: Vl-bert: Pretraining of generic visual-linguistic representations

authors:
Alane Suhr
Stephanie Zhou
Ally Zhang
Iris Zhang
Huajun Bai
Yoav Artzi

title: A corpus for reasoning about natural language grounded in photographs

venue: In Proceedings of the Annual Meeting of the Association for Computational Linguistics,

authors:
Hao Tan
Mohit Bansal

title: Lxmert: Learning cross-modality encoder representations from transformers

venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),

authors:
Hugo Touvron
Matthieu Cord
Matthijs Douze
Francisco Massa
Alexandre Sablayrolles
Hervé Jégou

title: Training data-efficient image transformers & distillation through attention

venue: arXiv preprint arXiv:2012.12877,

authors:
Xiaolong Wang
Ross Girshick
Abhinav Gupta
Kaiming He

title: Non-local neural networks

authors:
Ning Xie
Farley Lai
Derek Doran
Asim Kadav

title: Visual entailment task for visuallygrounded language learning

venue: arXiv preprint arXiv:1811.10582,

authors:
Yujia Xie
Xiangfeng Wang
Ruijia Wang
Hongyuan Zha

title: A fast proximal point method for computing exact wasserstein distance

venue: In Uncertainty in Artificial Intelligence,

authors:
Zhaoyang Zeng
Bei Liu
Jianlong Fu
Hongyang Chao
Lei Zhang

title: Wsod2: Learning bottom-up and top-down objectness distillation for weakly-supervised object detection

venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,

authors:
Jingzhao Zhang
Sai Praneeth Karimireddy
Andreas Veit
Seungyeon Kim
Sashank J Reddi
Sanjiv Kumar
Suvrit Sra

title: Why adam beats sgd for attention models

authors:
Wen Zhang
Yang Feng
Fandong Meng
Di You
Qun Liu

title: Bridging the gap between training and inference for neural machine translation

venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,

authors:
Sixiao Zheng
Jiachen Lu
Hengshuang Zhao
Xiatian Zhu
Zekun Luo
Yabiao Wang
Yanwei Fu
Jianfeng Feng
Tao Xiang
Philip HS Torr

title: Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers

venue: arXiv preprint arXiv:2012.15840,

authors:
Luowei Zhou
Hamid Palangi
Lei Zhang
Houdong Hu
Jason Corso
Jianfeng Gao

title: Unified vision-language pre-training for image captioning and vqa

heading: 1 Introduction

text:
Vision-Language Pre-training (VLP) has shown great benefits for Visual-Language (VL) tasks such as Visual Question Answering (VQA), Visual Entailment, etc., in many recent works [9, 21, 31, 32, 36, 40, 42, 51]. VLP is designed to learn vision and language (VL) joint representation and alignment with a huge number of image-text pairs. It provides a powerful initialization of multimodal representation for downstream VL tasks in a fine-tuning way. In this paper, we explore how to improve the multi-modal representation which is the key challenge in VLP from the perspective of intra-modal and inter-modal learning.
∗This work was performed when Hongwei Xue and Yupan Huang were visiting Microsoft Research as research interns.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Existing works use image features as visual tokens and learn their alignments with language tokens using a multi-modal Transformer. Three main types of image representation are commonly used in VLP: region feature, grid feature and patch projection. Most VLP models [9, 42, 40] extract region-based image features with an off-the-shelf object detector [34]. Each visual token in VLP corresponds to a pre-defined region feature. Some recent works directly learn grid features from images by CNNs to lift restrictions of bounding boxes and pre-defined object categories [22, 21]. They train CNN and multi-modal Transformer in an end-to-end fashion to enable visual features optimized for pre-training objectives. In addition to use CNNs for visual feature learning, a recent work ViLT [28] directly input image patch projections into the multi-modal Transformer to achieve the fastest inference speed with the lightest VLP architecture.
In vision-language (VL) tasks, the relation between visual concepts is crucial. For example, given a question “what is the man doing?” for an image of a surfing man, a VL model is expected to infer the relation of “surfing” from object “man” and object “surfboard”. However, existing three types of image representations fail to model the intra-vision relation. For both region feature and patch projection, each unit (i.e., bounding box or patch) is independent while global relations are not encoded in the visual embedding. For grid feature learned by CNNs, the local receptive field in the convolution layer results in local features of neighbor regions. Thus the two objectives of VLP, i.e., learning visual relation and inter-modal alignment, are encapsulated in the multi-modal Transformer network. Such design might restrict the inter-modal alignment learning in the Transformer by ignoring the specialized characteristic of each objective. On the other hand, for language, texts are highly structured and relations are explicitly provided by grammar. This inconsistent representation of different modalities distracts the multi-modal Transformer from the inter-modal alignment.
To better learn visual relation and further promote inter-modal alignment, we propose a fully Transformer VLP model which adopts self-attention for visual feature learning. Self-attention breaks the spatial inductive bias and enables long-range global relation learning of visual features. Thus, multi-modal Transformer can be specialized for multi-modal joint learning. In self-attention mechanism, each visual token is an approximate weighted mixture of all tokens. The higher weight indicates higher dependency. We name this way of image feature learning as visual parsing. As visual parsing provides dependencies of each visual token pair, inter-modality learning can be further promoted by masking visual tokens with high dependency, forcing the multi-modal Transformer to have more attention on the language side. To the best of our knowledge, our proposed Masked Feature Regression (MFR) is first ranking-based masking mechanism in VLP that leverages the intrinsic properties of visual extractor. To promote the inter-modality learning, we propose the Inter-Modality Flow (IMF) metric based on Attention Flow [1] to measure the fusion of inter-modality in VLP. IMF aims to quantify the information flow between the two modalities.
To verify the effectiveness of our approach, we conduct experiments on a wide range of visionlanguage tasks that are highly related to visual relation understanding and inter-modal reasoning. Our approach not only outperforms the state-of-the-art VLP models, but also shows superiority on the proposed IMF metric. We also conduct extensive ablation studies to demonstrate the effectiveness of our proposed self-attention visual parsing and parsing-based masking mechanism. We thoroughly probe the inter-modality learning in VLP from the perspective of information flow and data distribution. Our probing reveals how vision and language fuse with each other.
Our contributions in this paper are summarized as follows:
1 We are the first to adopt self-attention to learn visual features for VLP, aiming to promote intermodality learning in multi-modal Transformer. Our model outperforms existing works on a wide range of vision-language tasks.
2 We propose a novel Inter-Modality Flow (IMF) metric to measure and reveal vision and language fusion in VLP.
3 We design the first ranking-based masking mechanism for visual self-attention module to further promote inter-modality learning, verified by well-designed ablation studies.

heading: 2 Related Work

text:
Multi-Modal Pre-training. Pioneering works of Vision-Language Pre-training like ViLBERT [36] and LXMERT [42] adopt two-stream architecture, which utilizes two separate Transformers to encode image and text modalities, and a third Transformer for multi-modal fusion. Recent works using
single-stream architecture directly fuse the two modalities in a Transformer to automatically learn inter-and-intra modality fusion [40, 32, 31, 9]. The majority of existing works advocate the singlestream designs, which show stronger performance and include fewer parameters than two-stream models. For above reasons, we mainly focus on one-stream architecture in this paper.
Single-stream architectures typically employ Transformer to learn multi-modal contextualized features based on the singular embedding of each modality. For vision part, most VLP works use Bottom-Up and Top-Down attention [2] to extract region-level visual features by a Faster R-CNN [39] detector pre-trained on Visual Genome dataset [29]. Region-level features may miss the contextual information out of the bounding boxes and often suffer from low quality, noisy, and over-sampled boxes. To overcome the restrictions of region-based image features, SOHO [21] designs an end-toend VLP pipeline which directly learns image embedding at pixel-level by a CNN. This end-to-end learning paradigm does not require bounding box annotations and it enables inference 10 times faster than region-based approaches. A recent work ViLT [28] directly inputs patch projections into a multi-modal Transformer to achieve a fast inference speed. While ViLT has performance gaps with existing methods on some downstream tasks like VQA.
Vision Transformer. As convolutional network only captures information in a local window, it will miss correlations of long-range features. To improve the capability of encoding distant dependencies or heterogeneous interactions, some works complement CNNs by extending self-attention modules [17, 44]. The augmentation by self-attention also benefits general visual feature extraction. This kind of method has been applied for the object detection task [8, 52, 47].
The pioneering work of Vision Transformer (ViT) [13] totally abandons convolution and applies a Transformer architecture on image patch projections for image classification. Inspired by ViT, there are some recent works studying vision Transformer for a broad range of vision tasks such as image classification [35, 43], object detection [4], semantic segmentation [50]. In this paper we adopt [35] as our visual backbone. Since CNN and Transformer favor different kinds of optimizer [48], our fully Transformer structure makes it easier to train and fine-tune. We utilize unified optimizer for vision Transformer and multi-modal Transformer.

heading: 3 Approach

heading: 3.1 Self-Attention Visual Parser

text:
Despite grid features overcome the restrictions of region-based image features and keep all visual information in images [21], CNN-based image feature learning remains challenging when utilized to vision-language pre-training. Specifically, convolution network tends to focus on local regions in images, adding the learning of global visual relation to the multi-modal Transformer. Instead of concentrating on inter-modal interaction learning, multi-modal Transformer has to learn intra-model interactions in visual modality as well. For the language side, texts are highly structured and they explicitly provide some relations from grammar. For example, prepositions and verbs often indicate the relations of objects. Thus, the two modalities are fed into the multi-modal Transformer at different information levels. This inconsistency makes Transformer tend to learn stand-alone representation of each modality. As a result, the multi-modal Transformer is distracted from focusing on learning inter-modal alignment.
Inspired by Vision Transformer (ViT) [13], we apply self-attention for visual embedding in visionlanguage pre-training. To learn visual features, we adopt a vision Transformer V T with parameters θ:
V = V T (I, θ) ∈ Rm×c, (1)
where I is the input image. m denotes the number of visual features, and c is the dimension of hidden states. To reduce computational complexity, we adopt Swin Transformer [35], which has a hierarchical structure. Although shifted windows makes the receptive field smaller than the whole image, the large window size and gradual down-sampling mechanism make the spatial inductive bias much lower than CNN. We also explicitly get the correlations of visual tokens. In the last layer of vision Transformer, the correlation of each token pair can be modeled by attention weights which track how features get mixed.

heading: 3.2 Inter-modality Probing

text:
In single-stream structure for Vision-Language Pre-training, a multi-modal Transformer is used to fuse the vision and language modality. To quantify the interactions between two modalities, we propose a metric named Inter-Modality Flow (IMF) to probe how vision and language fuse together.
Transformer is a stack of self-attention modules and feed forward neural networks. The interactions of image-and-text are learnt in self-attention modules. Following Attention Flow [1], we first compute values in layer l after excluding shared operations on each tokens as:
Tl+1 = Tl +W l attTl, (2)
where Tl and Tl+1 are the input and output of layer l. W latt is layer l’s attention weight. To account for residual connections, weights of mixture can be represented as:
Al = 0.5(I +W latt). (3) To track down the information propagating in Transformer layers, we first calculate attention flows of arbitrary two layers, e.g., layer i and layer j where j ≥ i :
Ai,j = i∏ a=j Aa. (4)
This calculation sums over all possible paths between two layers, and Ai,j is the attention weight matrix of layer i’s input to layer j’s output. Specifically, when j = i, Ai,i represents attention weights in layer i. To measure the interactions between two modalities, we propose an Inter-Modality Flow F i,jinter and it is computed as the proportion of inter-modal attention among all attentions:
F i,jinter = A i,j inter/(A i,j inter +A i,j intra), (5)
where Ai,jinter is the summation over all attentions of visual and language inter-modality features, while Ai,jintra sums over one modality, vision or language:
Ai,jinter = ∑
ϕinter
Ai,jx,y; A i,j intra = ∑ ϕintra Ai,jx,y, (6)
where ϕinter = {x ∈ V, y ∈ L or x ∈ L, y ∈ V } and ϕintra = {x, y ∈ V or x, y ∈ L}. L and V are token index set of vision and text. We calculate F i,jinter on pre-training data to probe how the two modalities interact with each other.

heading: 3.3 Pre-training Pipeline

text:
Model Overview. Figure 1 shows the overview of our vision-language pre-training framework. Our model is composed of a vision Transformer and a multi-modal Transformer. The vision Transformer takes an image as input and outputs the visual tokens V = {v1, v2, ..., vi, ...vm}. To encode spatial information of images, we utilize 2-D position embedding computed by sine function following other works [8, 37, 13]. We apply a Linear layer and Layer Normalization [3] to embed the vision tokens. For the input sentence, we follow BERT [11] to tokenize and get word embeddings W . We concatenate vision and language tokens (V and W) to form an input sequence for multi-modal learning. Similar to other VLP models, we add two special tokens [CLS] and [SEP] into the input sequence to indicate classification position and the separation of two modalities, respectively. A multilayer Transformer is adopted to take the joint vision-language input, and outputs the attended features. In order to distinguish it from the vision Transformer V T , we call it multi-modal Transformer MT .
We adopt three pre-training tasks in our model: Masked Language Modeling (MLM), Image-Text Matching (ITM) and Masked Feature Regression (MFR). Among them, MLM and ITM are two commonly used pre-training task and MFR is a novel task which is proposed to mask visual tokens with similar or correlated semantics in our framework.
Masked Language Modeling. We adopt Masked Language Modeling (MLM) following most vision-language pre-training works [9, 21, 28] to predict the ground-truth labels of masked text tokens from the contextualized tokens:
LMLM = −E(W,V) log p ( wi | W\i,V ) . (7)
We adopt the same masking strategy as in BERT, and use a Linear layer as the MLM head to output logits over vocabulary, which are then computed as the negative log likelihood loss for the masked tokens.
Image-Text Matching. To learn the inter-modal alignment, we adopt Image-Text Matching (ITM) task for pre-training as in most vision-language pre-training works [9, 21, 28]. With the probability of 0.5, we randomly replace the aligned image to a different image. We use a single linear layer as ITM head to predict logits y over binary class (match or not), and we compute negative log likelihood loss as our ITM loss:
LITM = −E(W,V) log p(y | W,V). (8)
We also design a vision-language token alignment (VLA) task inspired by the word region alignment objective in [9, 28]. Our VLA loss optimize the Optimal Transport (OT) distance approximated by IPOT [46]. Following [9, 28], we add the VLA loss multiplied by 0.1 to the ITM loss.
Masked Feature Regression. Without bounding boxes, random masking for visual feature regression loses its effectiveness as the model will directly copy from neighbor features [21]. By visual parsing introduced in Section 3.1, correlations of each visual token are explicitly modeled by attention weights. Similar to Al in Equation 3, we use the last layer’s attention weight for masking. We assume that visual tokens of high attentions share similar semantics or correlations. We first randomly pick one visual token to mask, and then mask tokens with top-k attention weights. We apply L2 regression between masked tokens and regressed features:
LMFR = k∑
i=1
∥∥∥v(i)m − r (v(i)m )∥∥∥2 2 . (9)
The vision Transformer and multi-modal Transformer are trained end-to-end with above objectives:
Lpre = LMLM + λ1LITM + λ2LMFR (10)

heading: 4 Experiments

heading: 4.1 Pre-training Details

text:
We follow the dataset settings of SOHO [21] for pre-training. We focus on in-domain datasets: MSCOCO Captions (MSCOCO) [33] and Visual Genome Dense Captions (VG) [29], which are typical in-domain datasets for many VL downstream tasks. When comparing with UNITER [9], we use its in-domain pre-training results for fairness.
We follow BERT to adopt the WordPiece tokenizer [49] to split a sentence into word tokens. We adopt Swin Transformer for vision Transformer and a 12-layer Transformer for multi-modal Transformer. If not specified, we use 384× 384 as input resolution. This resolution is much lower than the resolution of 600×1000 or 800×1333 adopted by most previous works [36, 9, 21, 28]. Increasing the resolution will likely further improve our performance as per the findings by [25]. Our models are initialized based on ImageNet [10] and BERT [11]. We use AdamW optimizers for vision Transformer with learning rate 5e-6 and multi-modal Transformer with learning rate 5e-5 empirically. Empirically, we set the k in Equation 9 as 7. We set the λ1 and λ2 in Equation 10 as 1 and 0.01 respectively. Our model is pre-trained on 8 NVIDIA Tesla V100 GPUs with a batch size of 2048. Following SOHO [21], the learning rate is warmed up for the first 500 iterations. The training process takes 40 epochs until convergence and the learning rate decays by 10 times at 25th and 35th epoch. To reduce memory cost, we pair an image with four texts in each batch, including two matched pairs and two unmatched pairs. MLM and MFR tasks are only calculated with the matched image-text pairs.

heading: 4.2 Downstream Tasks

text:
We evaluate our method by fine-tuning the pre-trained model on downstream vision-language tasks. As we particularly focuses on visual relation and inter-modal learning, we choose four tasks related to visual relation understanding and inter-modal reasoning: Image-Text Retrieval, Visual Question Answering (VQA), natural language for visual reasoning (NLVR), and fine-grained visual reasoning (Visual Entailment). We compare our model with several task-specific and pre-training models. Among them, SOHO [21] and ViLT [28] are end-to-end VLP models like ours. SOHO, ViLT and our model adopt CNN, Linear projection and Transformer for visual embedding learning, respectively.
Image-Text Retrieval. Image-text retrieval aims to retrieve the most relevant text from candidate images, or vice versa. Image-text retrieval includes two sub-tasks: image-to-text retrieval (TR) and text-to-image retrieval (IR). We follow the same practice as SOHO to conduct image-text retrieval for fair comparisons. During training, we construct image-text pairs in a mini-batch by sampling aligned pairs from ground-truth annotations, and unmatched pairs from other captions within the mini-batch. To predict whether an image-text pair is aligned or not, we use the joint embedding representation of the [CLS] token from Transformers to perform binary classification. Since the binary classification objective of image-text retrieval model is consistent with the image-text matching (ITM) task in pre-training stage, we initialize the task-specific head from the pre-trained ITM head for better initialization. We adopt AdamW optimizer with a learning rate of 5e-5. The mini-batch size is set to 32. We train 10 epochs until convergence and decay the learning rate by half at 5th epoch empirically.
Experiment results on Flickr30k [38] are shown in Table 1. Our model outperforms ViLT and SOHO under all metrics on Flickr30k. The promising results of our model on image-text retrieval indicate the advantage of our fully Transformer architecture for learning cross-modal alignment.
Visual Question Answering. Visual Question Answering (VQA) aims to give an answer for a question and an image. VQA task is typically formulated as a classification problem according to the majority of existing works. We adopt a multi-layer perception head to output logits from the [CLS] token. Following [27], we optimize the model by a binary cross-entropy loss on 3,192-way classifications. We fine-tune for 100 epochs with a batch size of 512 until convergence. We keep the optimizer the same as in the pre-training stage, and we decay the learning rate by 10 at the 65th and 75th epochs empirically.
We present the experimental results on VQA v2.0 dataset in Table 2. Our model obtains 0.75% and 0.7% absolute gains on test-dev and test-std splits over SOHO respectively. The results validate the effectiveness of modeling the intra-modality visual information with self-attention mechanism to facilitate intelligent visual question answering.
Visual Reasoning. Visual Reasoning with Natural Language (NLVR) aims to predict whether a text is related to a given pair of images. With the requirement of comparing two image-text pairs, NLVR further focuses on the compositional visual reasoning abilities of relations, comparisons, and quantities. For this task, we evaluate our model on NLVR2 dataset [41]. As there are two input images instead of one input image used in the pre-training setup, we follow the pair method introduced in UNITER [9] to input two image-text pairs to Transformer and get two embedding vectors from [CLS] tokens. Then we learn a classifier that takes the concatenation of the embedding vectors to infer “true” or “false” by a cross-entropy loss. We fine-tune for 80 epochs with a batch size of 512 until convergence. We decay the learning rate by 2 at the 60, 65, 70, 75, 78, 79th epochs empirically.
Results are shown in Table 3. We observe a clear improvement of our model over previous state-ofthe-art model SOHO, where 1.24% and 0.73% absolute accuracy improvements are obtained on the val and test splits respectively. Over 2% improvements are made over ViLT. Our promising results demonstrate our advantage in compositional visual reasoning by promoting inter-modal learning with a fully Transformer architecture.
Visual Entailment. Visual Entailment (VE) is a fine-grained visual reasoning task to infer whether an image semantically entails a text. To classify the more fine-grained relationship than NLVR between an image and a text pair, VE aims to infer the image-to-text relationship to be true (entailment), false (contradiction) or neutral. For this task, we evaluate our model on SNLI-VE dataset [45] which is constructed based on Stanford Natural Language Inference (SNLI) [6] and Flickr30K [38] datasets. We follow [9, 21] to perform the VE task as a three-way classification problem. The model predicts the scores of each class by a Linear layer on the representation output by the Transformer from the [CLS] token. We fine-tune the model for 25 epochs with a batch size of 512 until convergence. The learning rate is initialized as 5e-5, and decayed by 10 at the 15th, 20th and 23th epochs empirically.
Table 4 compares models on SNLI-VE dataset. Here we see that our model and SOHO are significantly better than UNITER with about 6% absolute gains on accuracy due to our end-to-end training architecture. Our method is comparable with SOHO on the val split and is slightly better than SOHO on the test split.

heading: 4.3 Ablation Study

text:
To validate the effectiveness of our visual parsing for vision-language pre-training, we conduct ablation studies on the VQA task. To compare CNN and Transformer on visual embedding, we adopt ResNet 50 and 101 [19] which respectively have comparable and much larger parameters and FLOPs compared with Swin-Tiny Transformer. For multi-modal Transformer, we use a 3-layer Transformer. We use the same pre-training and VQA settings as in Section 4.1, except for image resolution. We set resolution as 224× 224 in all ablation studies for fair comparison. We also study the proposed Masked Feature Regression (MFR) objective to verify the effectiveness of masking based on the parsing attention weights. Specifically, we design four settings: without MFR, random masking (MFRRand), masking based on parsing weights (MFR), and a variant (MFRCNN) designed for CNN which measures correlations by cosine distance. We set k in Equation 9 as visual tokens number m times the probability of random masking p: k = m ∗ p.
Results are presented in Table 5. The Inter-Modality Flows of all Swin-Tiny backbones are higher than ResNet 50 and ResNet 101. This shows that vision Transformer provides better visual embedding for inter-modality learning compared with CNNs. Although the parameters and FLOPs of the ResNet 101 backbone is more than 1.5 times of our Swin-Tiny Transformer backbone, the latter achieves better result by about 2% on the VQA score. Inter-Modality Flow verifies the superiority of self-attention visual embedding in promoting inter-modality learning. From this view, we further try to add a language-specific Transformer TLan which process the input language before feeding into crossmodal Transformer. From the results, TLan can also promote performances. For fair comparison. we do not include TLan in our final model.
To study the proposed Masked Feature Regression (MFR), the last two lines show that the masking based on the attention weights further promotes the interaction of two modalities. MFRCNN also shows improvement yet less significant than MFR. To intuitively demonstrate the attention-based masking, we sample some results and visualize the mask of visual tokens. Compared with random masking, attention-based masking tends to mask regions with similar semantics. In Figure 2, random masking lacks the ability of preventing the multi-modal Transformer from directly copying from neighbor regions. While attention-based masking mechanism forces the model to refer to the language side for inferring masked features.

heading: 4.4 Inter-modality Probing

text:
In this section, we aim to verify our model’s effectiveness of promoting inter-modality learning and better study how the two modalities fuse together. We choose UNITER [9], SOHO [21], ViLT [28] and our model as models that use region features, grid features, patch projections, features learned by self-attention, respectively. These four models all adopt a 12-layer Transformer for fusion. The fusion of two modalities can be demonstrated from two perspectives: the inter-modal interaction and feature distance of two modalities in the joint space. Thus we do the inter-modality probing by our proposed Inter-Modality Flow (IMF) and distance of data distribution measurement. We will discuss the difference between these two perspectives in this section.
Inter-Modality Flow. Inter-Modality Flow is proposed to quantify interaction between vision and language modalities. Majority of existing works on VLP fuse the vision and language modality
by a multi-modal Transformer. Our proposed Inter-Modality Flow measures the information flow between two modalities from layer i’s input to layer j’s output. A larger information flow implies more interactions between image and text. We design two Inter-Modality Flow probing tasks layer by layer in the multi-modal Transformer. We calculate the F 1,jinter, j ∈ [1, 12] to track the process of inter-modality learning and F i,iinter, i ∈ [1, 12] to probe each layer’s impact on modality fusion. From the result shown in Figure 3, our model has larger IMF from the input tokens to every layers consistently. This indicates that there are more interactions between image and text in our model. SOHO and UNITER have comparable IMF. ViLT has much smaller IMF than other methods, as patch projections learn little about visual relationships. The result shown in Figure 4 indicates that the interactions gradually increase in deeper layers. It is worth noting that the interactions decrease a little in the last few layers as each modality optimizes by different pre-training tasks such as MLM and MFR. For the models in these two probing tasks, the IMF is positively correlated with the performance of downstream tasks, which testify to the importance of inter-modality learning in VLP.
Data Distribution. Following [7], we study the multi-modal fusion degree of a model from the view of data distribution. In every layer, we extract all visual and language features and apply the k-means algorithm (with k = 2) on features to partition them into two clusters. We measure the difference between the k-means clusters and ground-truth visual/textual clusters via Normalized Mutual Information (NMI) which is an unsupervised metric for evaluating differences between clusters. A larger NMI value implies more significant distinction between two clusters, indicating a lower fusion degree [24]. We average all NMI scores on data samples from Visual Genome.
From Figure 5, SOHO and our model has lower NMI scores than UNITER in every layer consistently. One explanation is that end-to-end training of visual backbone reduces the distinction between two modalities’ data distributions. The NMI scores of UNITER gradually decrease while SOHO and our model’s NMI score fluctuate within small values. In ViLT, layers with large and small NMI both exist. NMI of ViLT is overall larger than the other three models as it lacks visual backbone to optimize visual features towards alignment to word embeddings. As NMI only evaluates differences between data distributions, it lacks the capacity of measuring inter-modal interactions. Thus, our proposed Inter-Modality Flow is better correlated to inter-modal alignment.

heading: 5 Conclusion

text:
In this paper, we explore the benefit of self-attention for visual feature learning in Vision-Language Pre-training (VLP). We explore CNN’s limitation in intra-vision modality learning by introducing fully Transformer for visual embedding in VLP. We also design a novel masking optimization mechanism named Masked Feature Regression (MFR) in Transformer to further promote the intermodality learning. To verify MFR and probe the mechanism of inter-modal alignment, we propose Inter-Modality Flow (IMF) to measure the interaction between vision and language modalities. In the future, we will further explore the different properties of various modalities and design a unified framework for inter-modality interaction [20]. Another potential direction is to involve more than two modalities in inter-modality learning to enable more powerful understanding and interactions, such as image tags [16, 15].
Acknowledgments and Disclosure of Funding
Funding in direct support of this work: NSF of China under Grant 61836011 and 62021001, GPUs provided by Microsoft Research Asia. Additional revenues related to this work: Internship at Microsoft Research Asia.

heading: A Experiment Statistics

text: We summarizes the statistics of all our pre-training and downstream tasks in Table 7.

heading: B Model Size

text:
As performances are strongly related to image size and Transformer token number which affect FLOPs, we consider FLOPs as model size. we list FLOPs of models in detail:
From the results of above table, we can see that our model has much fewer FLOPs than SOHO and region-based models.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In our supplementary material. (c) Did you discuss any potential negative societal impacts of your work? [Yes] In our
supplementary material. (d) Have you read the ethics review guidelines and ensured that your paper conforms to
them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] (b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] (c) Did you include any new assets either in the supplemental material or as a URL? [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating? [No] (e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [No] 5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]

title: Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training