summaries: The paper presents a new saliency map interpretability method for the task of image classification. It considers the saliency map as a random variable and computes the posterior distribution over it. The likelihood measures the predictions of the classifier for an image and its perturbed counterpart. The prior encodes positive correlation among adjacent pixels. Variational approximation is used to approximate the posterior.

summaries: This paper proposed a method for generating saliency maps for image classifiers that are stochastic (instead of deterministic). The probabilistic model assumes a saliency map random variable that generates the data with a classifier. The inference is done by variational methods. The paper presents several qualitative examples and a comparison to previous work using the pixel perturbation benchmark.

summaries: This paper proposes a new interpretability method for image classification networks. It considers a saliency map as a random variable and aims to calculate the posterior distribution over the saliency map.  The likelihood function and the prior distribution are then designed to make the posterior distribution over the saliency map explain the behavior of the classifier’s prediction. Quantitative evaluation on the perturbation benchmark as well as qualitative result show the effectiveness of the proposed method over baselines.

abstractText: Saliency maps have been widely used to explain the behavior of an image classifier. We introduce a new interpretability method which considers a saliency map as a random variable and aims to calculate the posterior distribution over the saliency map. The likelihood function is designed to measure the distance between the classifier’s predictive probability of an image and that of locally perturbed image. For the prior distribution, we make attributions of adjacent pixels have a positive correlation. We use a variational approximation, and show that the approximate posterior is effective in explaining the classifier’s behavior. It also has benefits of providing uncertainty over the explanation, giving auxiliary information to experts on how much the explanation is trustworthy.

authors:
Julius Adebayo
Justin Gilmer
Michael Muelly
Ian Goodfellow
Moritz Hardt
Been Kim

title: Sanity checks for saliency maps

venue: In Advances in Neural Information Processing Systems,

authors:
Marco Ancona
Enea Ceolini
Cengiz Öztireli
Markus Gross

title: Towards better understanding of gradient-based attribution methods for deep neural networks

venue: arXiv preprint arXiv:1711.06104,

authors:
Alejandro Barredo Arrieta
Natalia Dı́az-Rodrı́guez
Javier Del Ser
Adrien Bennetot
Siham Tabik
Alberto Barbado
Salvador Garcı́a
Sergio Gil-López
Daniel Molina
Richard Benjamins

title: Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai

venue: Information Fusion,

authors:
Sebastian Bach
Alexander Binder
Grégoire Montavon
Frederick Klauschen
Klaus-Robert Müller
Wojciech Samek

title: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation

venue: PloS one,

authors:
Chun-Hao Chang
Elliot Creager
Anna Goldenberg
David Duvenaud

title: Explaining image classifiers by counterfactual generation

venue: In International Conference on Learning Representations,

authors:
Jianbo Chen
Le Song
Martin Wainwright
Michael Jordan

title: Learning to explain: An information-theoretic perspective on model interpretation

venue: In International Conference on Machine Learning,

authors:
Piotr Dabkowski
Yarin Gal

title: Real time image saliency for black box classifiers

authors:
Ruth Fong
Mandela Patrick
Andrea Vedaldi

title: Understanding deep networks via extremal perturbations and smooth masks

venue: In Proceedings of the IEEE International Conference on Computer Vision,

authors:
Ruth C Fong
Andrea Vedaldi

title: Interpretable explanations of black boxes by meaningful perturbation

authors:
Dan Hendrycks
Thomas Dietterich

title: Benchmarking neural network robustness to common corruptions and perturbations

authors:
Irina Higgins
Loic Matthey
Arka Pal
Christopher Burgess
Xavier Glorot
Matthew Botvinick
Shakir Mohamed
Alexander Lerchner

title: beta-vae: Learning basic visual concepts with a constrained variational framework

venue: Iclr,

authors:
Matthew D Hoffman
David M Blei
Chong Wang
John Paisley

title: Stochastic variational inference

venue: The Journal of Machine Learning Research,

authors:
Sara Hooker
Dumitru Erhan
Pieter-Jan Kindermans
Been Kim

title: A benchmark for interpretability methods in deep neural networks

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Diederik P Kingma
Max Welling

title: Auto-encoding variational bayes

venue: arXiv preprint arXiv:1312.6114,

authors:
Alex Krizhevsky
Ilya Sutskever
Geoffrey E Hinton

title: Imagenet classification with deep convolutional neural networks. In Advances in neural information processing

authors:
Alexey Kurakin
Ian Goodfellow
Samy Bengio

title: Adversarial examples in the physical world

venue: arXiv preprint arXiv:1607.02533,

authors:
Scott M Lundberg
Su-In Lee

title: A unified approach to interpreting model predictions

venue: In Advances in neural information processing systems,

authors:
Grégoire Montavon
Sebastian Lapuschkin
Alexander Binder
Wojciech Samek
Klaus-Robert Müller

title: Explaining nonlinear classification decisions with deep taylor decomposition

venue: Pattern Recognition,

authors:
Adam Paszke
Sam Gross
Soumith Chintala
Gregory Chanan
Edward Yang
Zachary DeVito
Zeming Lin
Alban Desmaison
Luca Antiga
Adam Lerer

title: Automatic differentiation in pytorch

authors:
Marco Tulio Ribeiro
Sameer Singh
Carlos Guestrin

title: why should i trust you?” explaining the predictions of any classifier

venue: In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining,

authors:
Olga Russakovsky
Jia Deng
Hao Su
Jonathan Krause
Sanjeev Satheesh
Sean Ma
Zhiheng Huang
Andrej Karpathy
Aditya Khosla
Michael Bernstein

title: Imagenet large scale visual recognition challenge

venue: International journal of computer vision,

authors:
Wojciech Samek
Alexander Binder
Grégoire Montavon
Sebastian Lapuschkin
Klaus-Robert Müller

title: Evaluating the visualization of what a deep neural network has learned

venue: IEEE transactions on neural networks and learning systems,

authors:
Karl Schulz
Leon Sixt
Federico Tombari
Tim Landgraf

title: Restricting the flow: Information bottlenecks for attribution

authors:
Ramprasaath R Selvaraju
Michael Cogswell
Abhishek Das
Ramakrishna Vedantam
Devi Parikh
Dhruv Batra

title: Grad-cam: Visual explanations from deep networks via gradient-based localization

venue: In Proceedings of the IEEE international conference on computer vision,

authors:
Avanti Shrikumar
Peyton Greenside
Anshul Kundaje

title: Learning important features through propagating activation differences

venue: In Proceedings of the 34th International Conference on Machine Learning-Volume

authors:
Karen Simonyan
Andrew Zisserman

title: Very deep convolutional networks for large-scale image recognition

venue: arXiv preprint arXiv:1409.1556,

authors:
Karen Simonyan
Andrea Vedaldi
Andrew Zisserman

title: Deep inside convolutional networks: Visualising image classification models and saliency maps

venue: arXiv preprint arXiv:1312.6034,

authors:
Daniel Smilkov
Nikhil Thorat
Been Kim
Fernanda Viégas
Martin Wattenberg

title: Smoothgrad: removing noise by adding noise

venue: arXiv preprint arXiv:1706.03825,

authors:
Jost Tobias Springenberg
Alexey Dosovitskiy
Thomas Brox
Martin Riedmiller

title: Striving for simplicity: The all convolutional net

venue: arXiv preprint arXiv:1412.6806,

authors:
Suraj Srinivas
François Fleuret

title: Full-gradient representation for neural network visualization

authors:
Mukund Sundararajan
Ankur Taly
Qiqi Yan

title: Axiomatic attribution for deep networks

authors:
Yujia Xie
Hanjun Dai
Minshuo Chen
Bo Dai
Tuo Zhao
Hongyuan Zha
Wei Wei
Tomas Pfister

title: Differentiable top-k operator with optimal transport

venue: arXiv preprint arXiv:2002.06504,

authors:
Matthew D Zeiler
Rob Fergus

title: Visualizing and understanding convolutional networks

venue: In European conference on computer vision,

authors:
Luisa M Zintgraf
Taco S Cohen
Tameem Adel
Max Welling

title: Visualizing deep neural network decisions: Prediction difference analysis

venue: arXiv preprint arXiv:1702.04595,

heading: 1 INTRODUCTION

text:
Since the advent of deep learning brought significant improvement in general machine learning tasks (Krizhevsky et al., 2012), explaining deep networks have become an important issue (Ribeiro et al. (2016)). Problems inherent in training a deep neural network, such as fairness (Arrieta et al., 2020) or the model classifying based on unintended features (Ribeiro et al., 2016), can be mitigated when the model is finely explained. Therefore, the models that have gained users’ trust through explanation are preferred in practical applications.
Saliency maps, also called attribution maps or relevance maps, have been widely used for interpretability methods in classification tasks, typically in an image domain (Simonyan et al., 2013). A saliency map represents the importance of each feature of given data that influences the model’s decision. There have been several approaches for obtaining the saliency map, which are backpropagation based methods (Ancona et al., 2017; Bach et al., 2015; Lundberg & Lee, 2017; Montavon et al., 2017; Selvaraju et al., 2017; Shrikumar et al., 2017; Simonyan et al., 2013; Smilkov et al., 2017; Srinivas & Fleuret, 2019; Sundararajan et al., 2017) and perturbation based methods (Chang et al., 2019; Chen et al., 2018; Dabkowski & Gal, 2017; Fong et al., 2019; Fong & Vedaldi, 2017; Schulz et al., 2020; Zeiler & Fergus, 2014; Zintgraf et al., 2017). Regardless of the approaches, the common implicit assumption shared by most of the previous interpretability methods is that a saliency map exists in a deterministic manner when a model and an input data are given: one attribution map is provided to explain the model’s decision for each data point.
Instead of the implicit assumption, we propose a stochastic approach called Variational Saliency maps (VarSal) where it is assumed that the interpretation has inherent randomness. The intuition stems from the stochastic effect that makes interpretation methods more explainable. For instance, FIDO (Chang et al., 2019) expands the search space of the mask by drawing it from Bernoulli distribution. This approach prevents the mask to be searched in the local space when it is directly optimized (Fong & Vedaldi, 2017). The example informs us that the stochastic property draws better interpretation.
We define the posterior distribution as the probability of the saliency map when the training data and the classifier are given. To make the posterior behave as the distribution of explanation, it is essential to carefully design the likelihood function and the prior distribution. We follow the idea of perturbation based methods to form the likelihood where the input that only contains features which correspond to high attribution in a saliency map is likely to describe the classifier’s behavior. For modeling the prior, we propose a new covariance matrix of Gaussian distribution that implies the property of having a positive correlation among attributions of adjacent pixels. As this property
mimics total variation (TV) regularization, we name the prior as soft-TV Gaussian prior. After modeling the likelihood and the prior, the Variational Bayesian method (Hoffman et al., 2013; Kingma & Welling, 2013) is used since the posterior is intractable.
After the optimization, unlike most of perturbation based methods, VarSal produces a real-time saliency map since only a single forward pass is required for generating it. Also, the VarSal method provides high quality in the visual inspection where sophisticated borderlines exist with objectoriented attention. We compare VarSal with baseline methods on the perturbation benchmark test to show the effectiveness of our approach. At the end, we examine the benefit of employing a posterior distribution, which is uncertainty over the explanation.

heading: 2 RELATED WORK

text:
In this section, we take a look at perturbation based interpretability methods. Fong & Vedaldi (2017) optimize the cost function with respect to the mask which indicates the most important features in an image for the classifier’s prediction. This approach is further developed by Fong et al. (2019) where they introduce a new method for making a perturbed image which helps to reduce hyper-parameters and produces better qualitative results. Both methods should optimize the mask every time they receive input, which is computationally expensive. Dabkowski & Gal (2017) relax the problem of time complexity by using a trained network of which the output is a saliency mask. However, all three methods have a limitation for producing importance ranking among features of a given image since their objective is to produce a binary mask.
PDA (Zintgraf et al., 2017) produces a saliency map from a different perspective. It computes the importance of each pixel by regarding it as an unobserved pixel and marginalizes it out to get the predictive probability output of the classifier. The same idea is used in FIDO (Chang et al., 2019) to generate a perturbed image that is regarded as a sample from training data distribution. It optimizes the parameters of a Bernoulli dropout distribution for making a saliency mask. It helps exploring the search space of binary mask rather than being limited to local search since the mask is sampled from the distribution for each training iteration. Our method is similar to FIDO in that VarSal also explores the search space by sampling the saliency map from the encoder in the training phase.
There is an information theoretic approach for explaining the classifier’s prediction. Schulz et al. (2020) adopt an information bottleneck for restricting the flow of information in an intermediate layer by adding noise. They find the importance of each feature by calculating the information flow. Chen et al. (2018) also adopt mutual information concept and optimize its variational bound for training a network that maps an input image to a saliency map. VarSal is similar in that we also train the encoder network by optimizing the evidence lower bound (ELBO). However, our method differs in that we regard the saliency map as a random variable and aim to calculate the posterior over the saliency map.
3 VARIATIONAL SALIENCY MAPS
In this section, we introduce details of the VarSal method which provides stochastic saliency maps. Let us define a pre-trained classifier that we aim to interpret as M : Rc×h×w → Y where x ∈ Rc×h×w is an input with c, h, and w to be channel, height, and width of the input image, respectively, and Y = {1, 2, . . . ,K} is a set of classes. The classifier M provides categorical probability PM (·) = ŷ ∈ 4K−1 where 4K−1 is a K−1 simplex. Since the purpose of a saliency map s ∈ Rh×w is to describe the behavior of the classifier’s prediction, our goal is to calculate the posterior distribution of the saliency map, p(s|x,ŷ) (solid lines in Figure 1). By Bayes’ rule, the posterior is stated as:
p(s|x,ŷ) = p(ŷ|x,s) p(s|x) / Z , (1)
where Z is the marginal likelihood. To calculate the posterior, we should model two terms: the likelihood p(ŷ|x,s) and the prior p(s|x).

heading: 3.1 MODELING LIKELIHOOD

text:
The likelihood should be well-designed to make the posterior over the saliency map explain the behavior of the classifier. We focus on the property that the importance of each feature in an image is determined by observing the response of the classifier’s output when the feature is perturbed (Zeiler & Fergus, 2014; Fong & Vedaldi, 2017; Zintgraf et al., 2017). More specifically, important features are enough to correctly classify the input as target class with high confidence. This concept is first introduced by Dabkowski & Gal (2017), and called smallest sufficient region (SSR). The difference between SSR and our approach is that we do not consider the smallest sufficient region, but rather rank the features (therefore, s ∈ Rh×w, not s ∈ {0, 1}h×w). Moreover, we consider not the target class but the categorical probability to interpret the model itself. The likelihood is designed such that the aforementioned properties satisfy
− log p(ŷ|x,s,k) = DKL[PM (x) ‖ PM (x τ (k)(s)) ] + const, (2)
p(ŷ|x,s) = Ep(k)[ p(ŷ|x,s,k) ] , (3)
where DKL is a Kullback-Leibler (KL) divergence, τ (k) is a top-k operation, and is a perturb operation that makes local perturbation of input x. The top-k operation applied to the saliency map, τ (k)(s) ∈ {0, 1}h×w, acts as a mask where [τ (k)(s)]i,j = 1 when si,j corresponds to the biggest k attributions in s. This way, the top-k operation makes the conditional likelihood in equation 2 to consider only the selected features in the input. By varying k, the amount of selected features is controlled, and we set p(k) as uniform distribution. To make the local perturbation of input x using perturb operation , we follow the method proposed by Fong & Vedaldi (2017), x m = x ◦m + x̃ ◦ (1 −m), where x̃ is a baseline input, and ◦ is a pointwise multiplication. We bring three baseline settings in our experiment: blurred baseline1, noise baseline2, and mean baseline3.
The equation 2 states that the categorical probability ŷ is more likely when the distance between the classifier’s predictive probability of input x and that of perturbed input is close for given k. This makes sense since better saliency map that explains the classifier’s behavior would approximate the model’s prediction closer with the selected features of top-k attributions. Also, we do not consider the ground-truth class or the top-1 predicted class, but rather whole classes with predictive probability in order to examine the classifier’s behavior itself. To consider various values of k, we also take the expectation in equation 3.

heading: 3.2 SOFT-TV GAUSSIAN PRIOR

text:
The easiest way to model the prior distribution p(s|x) is to consider it as independent of x, p(s|x) = p(s), and design it as standard Gaussian distribution N (vec(s);0, I) where vec(s) ∈ Rhw is a vectorized version of s. However, the standard Gaussian prior does not consider the belief over the saliency map that the attribution of neighbor pixels might have correlation (Fong et al., 2019). Therefore, we propose a new prior distribution that expresses the belief. While Dabkowski & Gal (2017); Fong & Vedaldi (2017) proposed total variation (TV) regularization to prevent the saliency mask from being occurred adversarial artifacts, we mimic the TV method in building the prior distribution to provide the positive correlation between neighbor attributions. To be more specific, we design the prior distribution as zero mean Gaussian distribution,N (vec(s);0,Σ), and infuse the TV knowledge to the covarianace matrix by setting Σi,j > 0 when pixel i and pixel j are identical or adjacent. We have
Σi,j =  1 , if i=j α , if i−j ∈ {−w,−1, 1, w} and j ∈ Adji α2 , if i−j ∈ {−w−1,−w+1, w−1, w+1} and j ∈ Adji 0 , otherwise ,
(4)
1We define “blurred baseline” as an input image blurred with Gaussian kernel. 2The “noise baseline” is defined as Gaussian noise. 3We term “mean baseline” when the baseline is set to be the per channel mean of an original image and added by Gaussian noise.
where α > 0 and Adji is the adjacent index set of pixel i (Figure 9 in Appendix B). This way, we grant the TV knowledge to the Gaussian prior, and call it soft-TV Gaussian prior.

heading: 3.3 VARIATIONAL INFERENCE ON SALIENCY MAPS

text:
Modeling the likelihood as equation 3 and the prior as N (vec(s);0,Σ) makes the posterior of equation 1 intractable. Therefore, we approximate it with the distribution qθ(s|x) parameterized by θ (dotted lines in Figure 1) where the objective is to minimize the KL divergence between qθ(s|x) and p(s|x,ŷ):
argmin θ
DKL[ qθ(s|x) ‖ p(s|x,ŷ) ]
= argmin θ Eq[− log p(ŷ|x,s)︸ ︷︷ ︸ (∗) ] +DKL[ qθ(s|x) ‖ p(s|x) ]︸ ︷︷ ︸ (∗∗) . (5)
We apply a mean-field approximation with univariate Gaussian for each factorized term of approximate posterior, qθ(s|x) = N ( vec(s); µθ(x),diag(νθ(x)) ), where µθ(·) ∈ Rhw is the mean of the distribution and diag(νθ(·)) ∈ Rhw×hw is the diagonal covariance matrix with the main diagonal to be νθ(·) ∈ Rhw. µθ and νθ are collectively called encoder network parameterized by θ. The encoder network is optimized with the training dataset used for training the classifier M , and the reparameterization trick (Kingma & Welling, 2013) is applied during optimization. The schematic description is shown in Appendix A.
There are two problems in optimizing the equation 5: non-differentiable top-k operation in equation (∗) and computationally expensiveness in equation (∗∗). In case of (∗), the top-k operation τ (k) is non-differentiable where the gradient cannot flow backward. To overcome this issue, we approximate it with a differentiable SOFT operator proposed by Xie et al. (2020). This allows flowing the gradient from the classifierM to the encoder parameters θ. Note that the classifier M is a pre-trained classifier that we aim to interpret, and thus should be fixed.
Since the size of covariance Σ is large, which is hw×hw, it is computationally expensive to calculate (∗∗) when it is naively used (in case of Imagenet dataset (Russakovsky et al., 2015), the size of Σ is 2244 !). We solve the problem by decomposing Σ with a Kronecker product:
Σ = κh ⊗ κw , (6)
where⊗ is the Kronecker product, and κh ∈ Rh×h and κw ∈ Rw×w are tridiagonal matrices with 1 for main diagonal and α for first diagonal below and above the main diagonal (Figure 9 in Appendix B). After all, (∗∗) can be analytically derived as:
DKL[ qθ(s|x) ‖ p(s|x) ] = diag ( κ−1w )T · rsh (νθ) · diag (κ−1h )− sum (log νθ) + sum ( rsh (µθ) ( κ−1w · rsh (µθ) · ( κ−1h )T)) + const , (7)
where sum(·) is the summation of elements. For a vector b ∈ Rhw, we denote rsh(b) ∈ Rw×h as reshaping the vector b to the matrix where [rsh(b)]i,j = bi+wj . Also, for a square matrix B, diag(B) is the vector where the ith entry is Bi,i. The derivation of equation 7 is provided in Appendix D. The equation 7 shows that the computation is easily done in general deep learning frameworks such as Pytorch (Paszke et al., 2017).

heading: 4 EXPERIMENT

heading: 4.1 IMPLEMENTATION DETAIL

text:
There are two terms in the objective function equation 5: the reconstruction term (*) and the regularization term (**). The components in the regularization term have high dimension, which is hw. This is usually too large that the regularization term becomes dominant in the loss function.
We avoid this phenomenon by introducing a hyper-parameter β to the regularization term (Higgins et al., 2017). We set the default value of β to be 1/(10hw).
We test our method on the ImageNet dataset (Russakovsky et al., 2015). We use a pre-trained VGG16 (Simonyan & Zisserman, 2014) for the classifier M , and 16 convolutional layers for the encoder µθ and νθ. As for the variable k used in the top-k operator, we sample k from the uniform distribution, k ∼ U(hw/10, 9hw/10), for each training iteration. The baseline input x̃ is also randomly selected among three baselines (blurred, noise, and mean baseline) for each training iteration. As for α in the soft-TV Gaussian prior, we set α = 0.4. Finally, if not mentioned, the mean of approximate posterior is used when performing qualitative and quantitative experiments. More details are given in Appendix C.

heading: 4.2 QUALITATIVE RESULTS

text:
We perform visual inspection by comparing VarSal with previous interpretability methods. For fair comparison, with the data that are correctly classified, previous interpretability methods are performed based on the ground-truth target, while VarSal explains the predictive probability. In Figure 2, the first heatmap (Simonyan et al., 2013) are generated by the gradients of input. They visually highlights the object, but there exists sparsity. PDA (Zintgraf et al., 2017) usually provides unnecessary highlights since it only considers spatially local parts in the optimization process. Realtime saliency (Dabkowski & Gal, 2017) and EP (Fong et al., 2019) show a boolean mask for the saliency map with smoothed borderlines. This is because it optimizes the mask with the size smaller than the input, followed by performing upsampling for the final saliency mask. FIDO (Chang et al.,
2019) provides the shape of the object to some extent, but it still has sparsity. Compared to the previous approaches, VarSal does not contain upsampling process, but rather directly provides the saliency map with the same size of input image. As the last heatmap shows, the VarSal highlights the object with more sophisticated borderlines. More results are provided in Appendix E.
To investigate the importance of modeling the prior distribution, we perform visual inspection between the VarSal method trained with standard Gaussian prior and that of soft-TV Gaussian prior. As Figure 3 shows, VarSal trained with standard Gaussian prior provides saliency map with high frequency noise inside the object boundary. This is because the standard Gaussian prior does not constrain the adjacent attributions to have correlation. On the other side, VarSal optimized using the soft-TV Gaussian prior shows smaller variation of attribution between adjacent pixels. Moreover, the noise on the background has been reduced when the soft-TV prior is used for modeling the prior distribution.
4.3 SANITY CHECK
The prerequisite for becoming an interpretability method is to pass the sanity check (Adebayo et al., 2018). This is to identify whether the interpretability method provides a saliency map dependent of a classifier or a data, and is tested by randomizing the classifier’s parameters. The difference between saliency maps obtained by the original classifier M and the parameter-randomized classifier is measured by structural similarity index (SSIM) and Spearman rank correlation. It is known that Guided Back-
Prop (Springenberg et al., 2014) and Guided GradCAM (Selvaraju et al., 2017) do not pass the sanity check since the SSIM and the Spearman metric is close to 1. Instead, Figure 4 shows that our method has lower values for each metric and different randomization of the classifier, indicating that VarSal has passed the sanity check.

heading: 4.4 QUANTITATIVE RESULTS

text:
Determining the state-of-the-art interpretability method is challenging since there is no evaluation benchmark that exactly reflects the method’s performance (Hooker et al., 2019). Commonly used benchmarks can split the superiority and inferiority of each method to some extent, but cannot exactly rank them based on the quantitative evaluation indicator. We use pixel perturbation benchmark to verify the usefulness of our method.
For the pixel perturbation metric, image pixels are erased that correspond to the largest k% saliency values (Ancona et al., 2017; Samek et al., 2016) or the smallest k% saliency values (Srinivas & Fleuret, 2019), and observe the response to the change of classifier’s output. In our experiment, we erase pixels with the latter procedure as the former is more prone to create unnecessary artifacts that lead to misunderstanding of the reason for the score drop (Srinivas & Fleuret, 2019). We change pixel values of the input image to zero that correspond to the least k% values in the saliency map, and observe the KL divergence between the classifier’s predictive probability of original input and that of perturbed input. The interpretability method is thought to be better when the distance is smaller.
It takes expensive computational time for PDA and FIDO method, about 25 minute and 1 minute per one image (Figure 5(b)). Therefore, for evaluating PDA and FIDO, we randomly sample 100 data in the validation dataset to perform the top-k perturbation benchmark, with repeating the process 5 times. Other methods use entire validation dataset for the evaluation. For fair comparison, saliency map is generated from the predicted class for each interprebility methods except VarSal. Also, perturbing randomly drawn k pixels is suggested as a control experiment. We omit drawing error range for this control experiment since the error (standard deviation) is less than 1e-2. The results are shown in Figure 5(a). It is observed that Input-Gradient (Simonyan et al., 2013), IntegratedGradient (Sundararajan et al., 2017), and PDA gets close to the control setting as k gets larger. For others such as SmoothGrad-squared (Smilkov et al., 2017; Hooker et al., 2019), FIDO, and VarSal, they have similar values throughout the change of k values. Moreover, VarSal performs better when using the soft-TV Gaussian prior than using the standard Gaussian prior.
An interesting point is that the saliency map obtained by sampling from the approximate posterior instead of the mean values results in low-quality (red dotted line in Figure 5(a)). We speculate this is because the sampling method produces artifacts in a perturbed image that cause score drop (Kurakin et al., 2016). As shown in Figure 6, even though both the sampled saliency map and the mean saliency map captures objects with high values, the sampled one has high-frequency noise. When the image pixels corresponding to low k% of the noisy saliency map are perturbed, the perturbed image has the artifact of sharp color contrast between adjacent pixels that might cause score degradation.

heading: 4.5 UNCERTAINTY OVER EXPLANATION

text:
One advantage of having posterior distribution over the saliency map is that it gives us the uncertainty of the explanation. This is done by summarizing the posterior with its covariance matrix. Since the VarSal approximates the posterior with factorized Gaussian, we can observe the variance
of each attribution where the examples are shown in Figure 7(a). While the explanation (second row) has higher attribution at the object, the uncertainty over the explanation (third row) presents at the borderline of the object.
We also qualitatively compare the posterior results on the shifted samples. Recently Hendrycks & Dietterich (2018) have established Imagenet-C dataset that is created by visually corrupting the data in the Imagenet dataset. The first row in Figure 7(b) and (c) respectively shows the example of images corrupted by ”Brightness” and ”Motion blur” with level 5 of severity. While the explanation of corrupted images is similar to that of original images in that they all capture the object to some extent, the uncertainty of the explanation shows different appearance where the heatmaps of standard deviation are noisy in the corrupted images.

heading: 5 CONCLUSION AND FUTURE WORK

text:
In this paper, we presented a new perspective on a saliency map where it is assumed to be a random variable. After designing the likelihood function and the prior distribution that makes the posterior distribution over the saliency map explain the behavior of the classifier’s prediction, the approximate posterior is optimized by maximizing ELBO. The experimental results were performed with the mean of the approximate posterior, and showed that our method has visually sharp borderlines with object-oriented saliency map. For quantitative results, the pixel perturbation benchmark is used to prove the effectiveness of our method. We verified that using the proposed soft-TV Gaussian distribution rather than the standard Gaussian distribution for modeling the prior has better performance in both qualitative and quantitative comparison. It was also shown that the proposed VarSal method has a strong advantage over other methods in terms of inference computation complexity. Finally, we showed that our method provides not only the explanation but also the uncertainty over the explanation.
There remain future works for producing better quality of the posterior distribution over a saliency map. In modeling the likelihood, the problem of data distribution shift could be mitigated by generating a perturbed image that is expected to be sampled from training data distribution (Chang et al., 2019). It would also be interesting to consider axiomatic prior knowledge (Sundararajan et al., 2017; Srinivas & Fleuret, 2019) in modeling the prior distribution and the likelihood. Finally, the uncertainty consideration over explaination is believed to be critical since it can tells us how much the explanation given by an interpretability method can be trusted, which needs to be studied further.

heading: A SCHEMATIC DESCRIPTION

text:
Encoder (q)
𝝁(𝑋)
𝝂(𝑋)
Classifier ( )
y()*+,-+.
𝒩(0, Σ) Reconstruction LossRegularization Loss
Input data Saliency Mask
Sample Differentiable Top-k Operator
Fixed
y(3+4546
Figure 8: Schematic description.
An input image is fed into the encoder network that gives the mean and the variance of Guassian distribution (which is the approximate posterior). A saliency map is sampled from it, followed by passing a differentiable top-k operator to provide a binary mask. With this mask, the input image is perturbed. The perturbed image is passes a classifier M that gives categorical probability ŷperturb. The loss function is composed of two terms: the reconstruction term between ŷperturb and the categorical probability obtained from the original image ŷorigin, and the regularization term between the approximate posterior q(s|x) = N (s;µ(x),diag(ν(x))) and the prior distribution N (s;0,Σ). As the classifier M is the model that we aim to interpret, it is fixed so as the parameter not to be updated while training the encoder network.

heading: B SOFT-TV GAUSSIAN PRIOR

text:
We consider the prior knowledge that adjacent pixels in a saliency map have positive correlation (Figure 9(a)). After modeling the prior as Gaussian distribution N (s;0,Σ), the covariance matrix Σ is designed to infuse this prior knowledge into the prior distribution (Figure 9(b)). The covariance matrix is then decomposed by Kronecker product to better calculate the KL divergence of the regularization loss.
C IMPLEMENTATION DETAILS
Encoder architecture We use 17 convolution layers for the encoder network. To make the spatial size of the encoder’s input and output to be same, we do not use a pooling layer. Every convolution layer is comprised of a convolution with kernel size 3×3, stride 1, and padding 1, followed by batch normalization and a rectified linear unit. The number of output channels for each convolution is as follows: [64, 64, 64, 64, 32, 32, 32, 32, 32, 16, 16, 16, 16, 16, 16, 2]. The encoder network provides µ ∈ Rh×w and η ∈ Rh×w for each channel of the output where µ is the mean of the Gaussian distribution and η = log ν with ν ∈ Rh×w the variance of the Gaussian distribution.
hyper-parameters We use Adam (Kingma & Ba, 2014) optimizer with learning rate to be 0.0001, weight decay to be 0.0005, and betas to be (0.9, 0.99). We use batch size of 128 while training the encoder network. We run 10 epochs for the Imagenet dataset, and save the network that has the lowest loss.

heading: D PROOF OF REGULARIZATION LOSS EQUATION

text:
Let us first define the notation. ⊗ is the Kronecker product, and is the element-wise multiplication. sum(·) is the summation of elements. For a vector b ∈ Rhw, we denote rsh(b) ∈ Rw×h as reshaping
the vector b to the matrix where the (i, j)-th entry is b[i + wj], and diag(b) as the diagonal matrix where the diagonal is b. Also, for a square matrix B, diag(B) is the vector where the i-th entry is B[i, i]. For a matrix B, vec(B) denotes the vectorization by stacking the columns of the matrix B to a single column vector.
Recall that the approximate posterior is q(s|x) = N (µ,Σ0) where Σ0 = diag(ν) with µ,ν ∈ Rhw, and the prior distribution is p(s|x) = N (0,Σ1) where Σ1 = κh ⊗ κw with κh ∈ Rh×h and κw ∈ Rw×w. KL divergence between two Gaussian distribution is:
DKL[ q(s|x) ‖ p(s|x) ] = DKL[N (µ,Σ0) ‖ N (0,Σ1) ]
= 1
2
( tr ( Σ−11 Σ0 ) + µTΣ−11 µ− hw + log
|Σ1| |Σ0|
) . (8)
We compute each term in RHS of equation 8 to make it computationally efficient.
tr ( Σ−11 Σ0 ) = tr ( (κh ⊗ κw)−1 · diag (ν) ) = tr (( κ−1h ⊗ κ −1 w ) · diag (ν)
) = diag ( κ−1h ⊗ κ −1 w ) diag (ν)
= diag ( κ−1w )T · rsh (ν) · diag (κ−1h ) . (9) µTΣ−11 µ = µ T · ( κ−1h ⊗ κ −1 w ) · µ
= µT · ( κ−1h ⊗ κ −1 w ) · vec (rsh (µ))
= µT · vec ( κ−1h · rsh (µ) · κ −1 w T )
= sum ( rsh (µ) ( κ−1w · rsh (µ) · ( κ−1h )T)) . (10)
log |Σ1| |Σ0| = log |κh ⊗ κw| − log |diag (ν)|
= log |κh|w|κw|h − log hw∏ i=1 νi
= w · log |κh|+ h · log |κw| − log hw∏ i=1 νi
= −sum (log νi) + const . (11)
Therefore, the equation 8 is derived as:
DKL[ q(s|x) ‖ p(s|x) ] = diag ( κ−1w )T · rsh (ν) · diag (κ−1h ) + sum ( rsh (µ) ( κ−1w · rsh (µ) · ( κ−1h )T)) (12)
− sum (log νi) + const .
E QUALITATIVE RESULTS
bu st
ar d
pe li
ca n
tr il
ob it e ca pu ch
in li on fi
sh Ir is h_ w ol fh ou
nd m ee rk
at go ld fi
sh bu ll fr
og da m se lf ly
Image
A fr
ic an
_e le
ph an
t
Input Gradient PDA Real-Time EP FIDO VarSal (Ours)


summaries: This paper introduces a novel technique for debiasing pretrained contextual embedding models. Their approach trains a 2 layer fully-connected neural network which takes as input the output from the pretrained model and outputs a new, "debiased" representation. This model is trained by minimizing the InfoNCE between the representation produced of original sentence and the representation of that same sentence with some tokens replaced with differently-biased tokens (e.g. "his" -> "hers"). This paper also introduces a regularizer which minimizes the CLUB between the generated representation and a word embedding for a biased token.

summaries: This paper studied a debiasing method to remove social bias in pretrained NLP models. The authors proposed to train a neural network which takes the sentence representations of a pretrained NLP model as input and outputs the unbiased representations. The neural network is trained by maximizing the mutual information between a sentence and its “counterpart sentence”, which is automatically generated by replacing sensitive words by other values (e.g. replacing “he” with “she”). Moreover, the network can be further trained by minimizing the mutual information between the sentence representation and its sensitive word representation. The experiments show that the proposed method can effectively reduce bias while achieving better downstream task performance of the pretrained model.

summaries: The paper builds on recent working attempts to debais sentence encoders by considering modified sentences. Sentences are selected that, for example, contain gender cueing words, and then swap those words with a predetermined 'opposite' (i.e. man<->woman). The core novelty in the work is to train a lightweight modification the encoding of the sentence and its swap to (a) reduce the distance between the two embeddings, using a contrastive learning objective and (b) reduce the mutual information between cueing words and the new embeddings. Evaluated on a WHEAT style task, modified for sentences, the method performs significantly better than existing recent work.

abstractText: Pretrained text encoders, such as BERT, have been applied increasingly in various natural language processing (NLP) tasks, and have recently demonstrated significant performance gains. However, recent studies have demonstrated the existence of social bias in these pretrained NLP models. Although prior works have made progress on word-level debiasing, improved sentence-level fairness of pretrained encoders still lacks exploration. In this paper, we proposed the first neural debiasing method for a pretrained sentence encoder, which transforms the pretrained encoder outputs into debiased representations via a fair filter (FairFil) network. To learn the FairFil, we introduce a contrastive learning framework that not only minimizes the correlation between filtered embeddings and bias words but also preserves rich semantic information of the original sentences. On real-world datasets, our FairFil effectively reduces the bias degree of pretrained text encoders, while continuously showing desirable performance on downstream tasks. Moreover, our post hoc method does not require any retraining of the text encoders, further enlarging FairFil’s application space.

authors:
Alexander A Alemi
Ian Fischer
Joshua V Dillon
Kevin Murphy

title: Deep variational information bottleneck

venue: In International Conference on Learning Representations,

authors:
Su Lin Blodgett
Solon Barocas
Hal Daumé III
Hanna Wallach

title: Language (technology) is power: A critical survey

venue: of” bias” in nlp. arXiv preprint arXiv:2005.14050,

authors:
Piotr Bojanowski
Edouard Grave
Armand Joulin
Tomas Mikolov

title: Enriching word vectors with subword information

venue: Transactions of the Association for Computational Linguistics,

authors:
Tolga Bolukbasi
Kai-Wei Chang
James Y Zou
Venkatesh Saligrama
Adam T Kalai

title: Man is to computer programmer as woman is to homemaker? debiasing word embeddings

authors:
Shikha Bordia
Samuel R. Bowman

title: Identifying and reducing gender bias in word-level language models

venue: In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop,

authors:
Marc-Etienne Brunet
Colleen Alkalay-Houlihan
Ashton Anderson
Richard Zemel

title: Understanding the origins of bias in word embeddings

venue: In International Conference on Machine Learning,

authors:
Aylin Caliskan
Joanna J Bryson
Arvind Narayanan

title: Semantics derived automatically from language corpora contain human-like

venue: biases. Science,

authors:
Kaytlin Chaloner
Alfredo Maldonado

title: Measuring gender bias in word embeddings across domains and discovering new gender bias word categories

venue: In Proceedings of the First Workshop on Gender Bias in Natural Language Processing,

authors:
Ting Chen
Simon Kornblith
Mohammad Norouzi
Geoffrey Hinton

title: A simple framework for contrastive learning of visual representations

venue: In Proceedings of the 37th international conference on Machine learning,

authors:
Xi Chen
Yan Duan
Rein Houthooft
John Schulman
Ilya Sutskever
Pieter Abbeel

title: Infogan: Interpretable representation learning by information maximizing generative adversarial nets

authors:
Pengyu Cheng
Weituo Hao
Shuyang Dai
Jiachang Liu
Zhe Gan
Lawrence Carin

title: Club: A contrastive log-ratio upper bound of mutual information

authors:
Pengyu Cheng
Martin Renqiang Min
Dinghan Shen
Christopher Malon
Yizhe Zhang
Yitong Li
Lawrence Carin

title: Improving disentangled text representation learning with informationtheoretic guidance

venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,

authors:
Alexis Conneau
Douwe Kiela
Holger Schwenk
Loı̈c Barrault
Antoine Bordes

title: Supervised learning of universal sentence representations from natural language inference data

authors:
Jason V Davis
Brian Kulis
Prateek Jain
Suvrit Sra
Inderjit S Dhillon

title: Information-theoretic metric learning

venue: In Proceedings of the 24th international conference on Machine learning,

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova

title: Bert: Pre-training of deep bidirectional transformers for language understanding

venue: In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics,

authors:
Lucas Dixon
John Li
Jeffrey Sorensen
Nithum Thain
Lucy Vasserman

title: Measuring and mitigating unintended bias in text classification

venue: In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society,

authors:
Behnam Gholami
Pritish Sahu
Ognjen Rudovic
Konstantinos Bousmalis
Vladimir Pavlovic

title: Unsupervised multi-target domain adaptation: An information theoretic approach

venue: IEEE Transactions on Image Processing,

authors:
Aditya Grover
Jure Leskovec

title: node2vec: Scalable feature learning for networks

venue: In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining,

authors:
Michael Gutmann
Aapo Hyvärinen

title: Noise-contrastive estimation: A new estimation principle for unnormalized statistical models

venue: In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics,

authors:
Kaiming He
Haoqi Fan
Yuxin Wu
Saining Xie
Ross Girshick

title: Momentum contrast for unsupervised visual representation learning

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
R Devon Hjelm
Alex Fedorov
Samuel Lavoie-Marchildon
Karan Grewal
Phil Bachman
Adam Trischler
Yoshua Bengio

title: Learning deep representations by mutual information estimation and maximization

authors:
Mandar Joshi
Danqi Chen
Yinhan Liu
Daniel S Weld
Luke Zettlemoyer
Omer Levy

title: Spanbert: Improving pre-training by representing and predicting spans

authors:
Masahiro Kaneko
Danushka Bollegala

title: Gender-preserving debiasing for pre-trained word embeddings

venue: arXiv preprint arXiv:1906.00742,

authors:
Svetlana Kiritchenko
Saif Mohammad

title: Examining gender and race bias in two hundred sentiment analysis systems

venue: In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics,

authors:
Ryan Kiros
Yukun Zhu
Russ R Salakhutdinov
Richard Zemel
Raquel Urtasun
Antonio Torralba
Sanja Fidler

title: Skip-thought vectors. In Advances in neural information processing

title: Information theory and statistics

venue: Courier Corporation,

authors:
Keita Kurita
Nidhi Vyas
Ayush Pareek
Alan W Black
Yulia Tsvetkov

title: Measuring bias in contextualized word representations

authors:
Paul Pu Liang
Irene Mengze Li
Emily Zheng
Yao Chong Lim
Ruslan Salakhutdinov
LouisPhilippe Morency

title: Towards debiasing sentence representations

authors:
Zhouhan Lin
Minwei Feng
Cicero Nogueira dos Santos
Mo Yu
Bing Xiang
Bowen Zhou
Yoshua Bengio

title: A structured self-attentive sentence embedding

venue: arXiv preprint arXiv:1703.03130,

authors:
Yinhan Liu
Myle Ott
Naman Goyal
Jingfei Du
Mandar Joshi
Danqi Chen
Omer Levy
Mike Lewis
Luke Zettlemoyer
Veselin Stoyanov

title: Roberta: A robustly optimized bert pretraining approach

authors:
Laurens van der Maaten
Geoffrey Hinton

title: Visualizing data using t-sne

venue: Journal of machine learning research,

authors:
Sean MacAvaney
Andrew Yates
Arman Cohan
Nazli Goharian

title: Cedr: Contextualized embeddings for document ranking

venue: In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval,

authors:
Thomas Manzini
Yao Chong Lim
Yulia Tsvetkov
Alan W Black

title: Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings

authors:
Chandler May
Alex Wang
Shikha Bordia
Samuel R. Bowman
Rachel Rudinger

title: On measuring social biases in sentence encoders

venue: In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers),

authors:
Tomas Mikolov
Kai Chen
Greg Corrado
Jeffrey Dean

title: Efficient estimation of word representations in vector space

venue: arXiv preprint arXiv:1301.3781,

authors:
Aaron van den Oord
Yazhe Li
Oriol Vinyals

title: Representation learning with contrastive predictive coding

venue: arXiv preprint arXiv:1807.03748,

authors:
Sunghyun Park
Han Suk Shim
Moitreya Chatterjee
Kenji Sagae
Louis-Philippe Morency

title: Computational analysis of persuasiveness in social multimedia: A novel dataset and multimodal prediction approach

venue: In Proceedings of the 16th International Conference on Multimodal Interaction,

authors:
Matthew E Peters
Mark Neumann
Mohit Iyyer
Matt Gardner
Christopher Clark
Kenton Lee
Luke Zettlemoyer

title: Deep contextualized word representations

venue: In Proceedings of NAACL-HLT,

authors:
Soujanya Poria
Devamanyu Hazarika
Navonil Majumder
Gautam Naik
Erik Cambria
Rada Mihalcea

title: MELD: A multimodal multi-party dataset for emotion recognition in conversations

venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,

authors:
Rui Qian
Tianjian Meng
Boqing Gong
Ming-Hsuan Yang
Huisheng Wang
Serge Belongie
Yin Cui

title: Spatiotemporal contrastive video representation learning

venue: arXiv preprint arXiv:2008.03800,

authors:
Alec Radford
Karthik Narasimhan
Tim Salimans
Ilya Sutskever

title: Improving language understanding by generative pre-training, 2018

authors:
Victor Sanh
Lysandre Debut
Julien Chaumond
Thomas Wolf

title: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter

authors:
Dinghan Shen
Pengyu Cheng
Dhanasekar Sundararaman
Xinyuan Zhang
Qian Yang
Meng Tang
Asli Celikyilmaz
Lawrence Carin

title: Learning compressed sentence representations for ondevice text processing

authors:
Richard Socher
Alex Perelygin
Jean Wu
Jason Chuang
Christopher D Manning
Andrew Y Ng
Christopher Potts

title: Recursive deep models for semantic compositionality over a sentiment treebank

venue: In Proceedings of the 2013 conference on empirical methods in natural language processing,

authors:
Jiaming Song
Pratyusha Kalluri
Aditya Grover
Shengjia Zhao
Stefano Ermon

title: Learning controllable fair representations

venue: In The 22nd International Conference on Artificial Intelligence and Statistics,

authors:
Gabriel Stanovsky
Noah A Smith
Luke Zettlemoyer

title: Evaluating gender bias in machine translation

authors:
Fei Sun
Jun Liu
Jian Wu
Changhua Pei
Xiao Lin
Wenwu Ou
Peng Jiang

title: Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer

venue: In Proceedings of the 28th ACM International Conference on Information and Knowledge Management,

authors:
Jian Tang
Meng Qu
Mingzhe Wang
Ming Zhang
Jun Yan
Qiaozhu Mei

title: Line: Large-scale information network embedding

venue: In Proceedings of the 24th international conference on world wide web,

authors:
Michael V”olske
Martin Potthast
Shahbaz Syed
Benno Stein

title: TL;DR: Mining Reddit to learn automatic summarization

venue: In Proceedings of the Workshop on New Frontiers in Summarization,

authors:
Alex Wang
Amanpreet Singh
Julian Michael
Felix Hill
Omer Levy
Samuel Bowman

title: Glue: A multi-task benchmark and analysis platform for natural language understanding

venue: In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP,

authors:
Alex Warstadt
Amanpreet Singh
Samuel R Bowman

title: Neural network acceptability judgments

authors:
Kilian Q Weinberger
John Blitzer
Lawrence K Saul

title: Distance metric learning for large margin nearest neighbor classification

authors:
Svante Wold
Kim Esbensen
Paul Geladi

title: Principal component analysis

venue: Chemometrics and intelligent laboratory systems,

authors:
Tianyi Zhang
Varsha Kishore
Felix Wu
Kilian Q Weinberger
Yoav Artzi

title: Bertscore: Evaluating text generation with bert

heading: 1 INTRODUCTION

text:
Text encoders, which map raw-text data into low-dimensional embeddings, have become one of the fundamental tools for extensive tasks in natural language processing (Kiros et al., 2015; Lin et al., 2017; Shen et al., 2019; Cheng et al., 2020b). With the development of deep learning, largescale neural sentence encoders pretrained on massive text corpora, such as Infersent (Conneau et al., 2017), ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), and GPT (Radford et al., 2018), have become the mainstream to extract the sentence-level text representations, and have shown desirable performance on many NLP downstream tasks (MacAvaney et al., 2019; Sun et al., 2019; Zhang et al., 2019). Although these pretrained models have been studied comprehensively from many perspectives, such as performance (Joshi et al., 2020), efficiency (Sanh et al., 2019), and robustness (Liu et al., 2019), the fairness of pretrained text encoders has not received significant research attention.
The fairness issue is also broadly recognized as social bias, which denotes the unbalanced model behaviors with respect to some socially sensitive topics, such as gender, race, and religion (Liang et al., 2020). For data-driven NLP models, social bias is an intrinsic problem mainly caused by the unbalanced data of text corpora (Bolukbasi et al., 2016). To quantitatively measure the bias degree of models, prior work proposed several statistical tests (Caliskan et al., 2017; Chaloner & Maldonado, 2019; Brunet et al., 2019), mostly focusing on word-level embedding models. To evaluate the sentence-level bias in the embedding space, May et al. (2019) extended the Word Embedding Association Test (WEAT) (Caliskan et al., 2017) into a Sentence Encoder Association Test (SEAT). Based on the SEAT test, May et al. (2019) claimed the existence of social bias in the pretrained sentence encoders.
Although related works have discussed the measurement of social bias in sentence embeddings, debiasing pretrained sentence encoders remains a challenge. Previous word embedding debiasing methods (Bolukbasi et al., 2016; Kaneko & Bollegala, 2019; Manzini et al., 2019) have limited assistance to sentence-level debiasing, because even if the social bias is eliminated at the word level,
∗Equal contribution.
the sentence-level bias can still be caused by the unbalanced combination of words in the training text. Besides, retraining a state-of-the-art sentence encoder for debiasing requires a massive amount of computational resources, especially for large-scale deep models like BERT (Devlin et al., 2019) and GPT (Radford et al., 2018). To the best of our knowledge, Liang et al. (2020) proposed the only sentence-level debiasing method (Sent-Debias) for pretrained text encoders, in which the embeddings are revised by subtracting the latent biased direction vectors learned by Principal Component Analysis (PCA) (Wold et al., 1987). However, Sent-Debias makes a strong assumption on the linearity of the bias in the sentence embedding space. Further, the calculation of bias directions depends highly on the embeddings extracted from the training data and the number of principal components, preventing the method from adequate generalization.
In this paper, we proposed the first neural debiasing method for pretrained sentence encoders. For a given pretrained encoder, our method learns a fair filter (FairFil) network, whose inputs are the original embeddings of the encoder, and outputs are the debiased embeddings. Inspired by the multi-view contrastive learning (Chen et al., 2020), for each training sentence, we first generate an augmentation that has the same semantic meaning but in a different potential bias direction. We contrastively train our FairFil by maximizing the mutual information between the debiased embeddings of the original sentences and corresponding augmentations. To further eliminate bias from sensitive words in sentences, we introduce a debiasing regularizer, which minimizes the mutual information between debiased embeddings and the sensitive words’ embeddings. In the experiments, our FairFil outperforms Sent-Debias (Liang et al., 2020) in terms of the fairness and the representativeness of debiased embeddings, indicating our FairFil not only effectively reduces the social bias in the sentence embeddings, but also successfully preserves the rich semantic meaning of input text.

heading: 2 PRELIMINARIES

text:
Mutual Information (MI) is a measure of the “amount of information” between two variables (Kullback, 1997). The mathematical definition of MI is
I(x;y) := Ep(x,y) [ log p(x,y)
p(x)p(y)
] , (1)
where p(x,y) is the joint distribution of two variables (x,y), and p(x), p(y) are respectively the marginal distributions of x,y. Recently, mutual information has achieved considerable success when applied as a learning criterion in diverse deep learning tasks, such as conditional generation (Chen et al., 2016), domain adaptation (Gholami et al., 2020), representation learning (Chen et al., 2020), and fairness (Song et al., 2019). However, the calculation of exact MI in (1) is well-recognized as a challenge, because the expectation w.r.t p(x,y) is always intractable, especially when only samples from p(x,y) are provided. To this end, several upper and lower bounds have been introduced to estimate the MI with samples. For MI maximization tasks (Hjelm et al., 2018; Chen et al., 2020), Oord et al. (2018) derived a powerful MI estimator, InfoNCE, based on noise contrastive estimation (NCE) (Gutmann & Hyvärinen, 2010). Given a batch of sample pairs {(xi,yi)}Ni=1, the InfoNCE estimator is defined with a learnable score function f(x,y):
INCE := 1
N N∑ i=1 log exp(f(xi,yi)) 1 N ∑N j=1 exp(f(xi,yj)) . (2)
For MI minimization tasks (Alemi et al., 2017; Song et al., 2019), Cheng et al. (2020a) introduced a contrastive log-ratio upper bound (CLUB) based on a variational approximation qθ(y|x) of conditional distribution p(y|x):
ICLUB := 1
N N∑ i=1 [ log qθ(yi|xi)− 1 N N∑ j=1 log qθ(yj |xi) ] . (3)
In the following, we use the above two MI estimators to induce the sentence encoder, eliminating the biased information and preserving the semantic information from the original raw text.

heading: 3 METHOD

text:
SupposeE(·) is a pretrained sentence encoder, which can encode a sentence x into low-dimensional embedding z = E(x). Each sentence x = (w1, w2, . . . , wL) is a sequence of words. The embedding space of z has been recognized to have social bias in a series of studies (May et al., 2019; Kurita
et al., 2019; Liang et al., 2020). To eliminate the social bias in the embedding space, we aim to learn a fair filter network f(·) on top of the sentence encoder E(·), such that the output embedding of our fair filter d = f(z) can be debiased. To train the fair filter, we design a multi-view contrastive learning framework, which consists of three steps. First, for each input sentence x, we generate an augmented sentence x′ that has the same semantic meaning as x but in a different potential bias direction. Then, we maximize the mutual information between the original embedding z = f(x) and the augmented embedding z′ = f(x′) with the InfoNCE (Oord et al., 2018) contrastive loss. Further, we design a debiasing regularizer to minimize the mutual information between d and sensitive attribute words in x. In the following, we discuss these three steps in detail.

heading: 3.1 DATA AUGMENTATIONS WITH SENSITIVE ATTRIBUTES

text: We first describe the sentence data augmentation process for our FairFil contrastive learning. Denote a social sensitive topic as T = {D1,D2, . . . ,DK}, where Dk (k = 1, . . . ,K) is one of the potential bias directions under the topic. For example, if T represents the sensitive topic “gender”, then T consists two potential bias directions {D1,D2} = {“male”,“female”}. Similarly, if T is set as the major “religions” of the world, then T could contain {D1,D2,D3,D4} = {“Christianity”, “Islam”, “Judaism”, “Buddhism”} as four components. For a given social sensitive topic T = {D1, . . .DK}, if a word w is related to one of the potential bias direction Dk (denote as w ∈ Dk), we call w a sensitive attribute word of Dk (also called bias attribute word in Liang et al. (2020)). For a sensitive attribute word w ∈ Dk, suppose we can always find another sensitive attribute word u ∈ Dj , such that w and u has the equivalent semantic meaning but in a different bias direction. Then we call u as a replaceable word of w in direction Dj , and denote as u = rj(w). For the topic “gender” = {“male”, “female”}, the word w = “boy” is in the potential bias direction D1 = “male”; a replaceable word of “boy” in “female” direction is r2(w) = “girl” ∈ D2. With the above definitions, for each sentence x, we generate an augmented sentence x′ such that x′ has the same semantic meaning as x but in a different potential bias direction. More specifically, for a sentence x = (w1, w2, . . . , wL), we first find the sensitive word positions as an index set P , such that each wp (p ∈ P) is a sensitive attribute words in direction Dk. We further make a reasonable assumption that the embedding bias of direction Dk is only caused by the sensitive words {wp}p∈P in x. To sample an augmentation to x, we first select another potential bias direction Dj , and then replace all sensitive attribute words by their replaceable words in the direction Dj . That is, x′ = {v1, v2, . . . , vL}, where vl = wl if l /∈ P , and vl = rj(wl) if l ∈ P . In Table 1, we provide an example for sentence augmentation under the “gender” topic.

heading: 3.2 CONTRASTIVE LEARNING FRAMEWORK

text:
After obtaining the sentence pair (x,x′) with the augmentation strategy from Section 3.1, we construct a contrastive learning framework to learn our debiasing fair filter f(·). As shown in the Figure 1(a), our framework consists of the following two steps:
(1) We encode sentences (x,x′) into embeddings (z, z′) with the pretrained encoder E(·). Since x and x′ have the same meaning but different potential bias directions, the embeddings (z, z′) will have different bias directions, which are caused by the sensitive attributed words in x and x′.
(2) We then feed the sentence embeddings (z, z′) through our fair filter f(·) to obtain the debiased embedding outputs (d,d′). Ideally, d and d′ should represent the same semantic meaning without social bias. Inspired by SimCLR (Chen et al., 2020), we encourage the overlapped semantic information between d and d′ by maximizing their mutual information I(d;d′). However, the calculation of I(d;d′) is practically difficult because only embedding samples of d and d′ are available. Therefore, we use the InfoNCE mutual information estimator (Oord et al., 2018) to minimize the lower bound of I(d;d′) instead. Based on a learnable score function g(·, ·),
the contrastive InfoNCE estimator is calculated within a batch of samples {(di,d′i)}Ni=1:
INCE = 1
N N∑ i=1 log exp(g(di,d ′ i)) 1 N ∑N j=1 exp(g(di,d ′ j)) . (4)
By maximize INCE, we encourage the difference between the positive pair score g(di,d′i) and the negative pair score g(di,d′j), so that di can share more semantic information with d ′ i than other embeddings d′j 6=i.

heading: 3.3 DEBIASING REGULARIZER

text:
Practically, the contrastive learning framework in Section 3.2 can already show encouraging debiasing performance (as shown in the Experiments). However, the embedding d can contain extra biased information from z, that only maximizing I(d;d′) fails to eliminate. To encourage no extra bias in d, we introduce a debiasing regularizer which minimizes the mutual information between embedding d and the potential bias from embedding z. As discussed in Section 3.1, in our framework the potential bias of z is assumed to come from the sensitive attribute words in x. Therefore, we should reduce the bias word information from the debiased representation d. Let wp be the embedding of a sensitive attribute word wp in sentence x. The word embedding wp can always be obtained from the pretrained text encoders (Bordia & Bowman, 2019). We then minimize the mutual information I(wp;d), using the CLUB mutual information upper bound (Cheng et al., 2020a) to estimate I(wp;d) with embedding samples. Given a batch of embedding pairs {(di,wp)}Ni=1, we can calculate the debiasing regularizer as:
ICLUB = 1
N N∑ i=1 [ log qθ(w p i |di)− 1 N N∑ j=1 log qθ(w p j |di) ] , (5)
where qθ is a variational approximation to ground-truth conditional distribution p(w|d). We parameterize qθ with another neural network. As proved in Cheng et al. (2020a), the better qθ(w|d) approximates p(w|d), the more accurate ICLUB serves as the mutual information upper bound. Therefore, besides the loss in (5), we also maximize the log-likelihood of qθ(w|d) with samples {(di,wpi )}Ni=1. Based on the above sections, the overall learning scheme of our fair filter (FairFil) is described in Algorithm 1. Also, we provide an intuitive explanation to the two loss terms in our framework. In Figure 1(b), the blue and red circles represent d and d′, respectively, in the embedding space. The intersection I(d;d′) is the common semantic information extracted from sentences x and x′, while the two shadow parts are the extra bias. Note that the perfect debiased embeddings lead to coincident circles. By maximizing INCE term, we enlarge the overlapped area of d and d′; by minimizing ICLUB, we shrink the biased shadow parts.
Algorithm 1 Updating the FairFil with a sample batch Begin with the pretrained text encoder E(·), and a batch of sentences {xi}Ni=1. Find the sensitive attribute words {wp} and corresponding embeddings {wp}. Generate augmentation x′i from xi, by replacing {wp} with {rj(wp)}. Encode (xi,x′i) into embeddings di = f(E(xi)),d ′ i = f(E(x ′ i)).
Calculate INCE with {(di,d′i)}Ni=1 and score function g. if adding debiasing regularizer then
Update the variational approximation qθ(w|d) by maximizing log-likelihood with {(di,wpi )} Calculate ICLUB with qθ(w|d) and {(di,wpi )}Ni=1. Learning loss L = −INCE + βICLUB.
else Learning loss L = −INCE. end if Update FairFil f and score function g by gradient descent with respect to L.

heading: 4 RELATED WORK

heading: 4.1 BIAS IN NATURAL LANGUAGE PROCESSING

text: Social bias has recently been recognized as an important issue in natural language processing (NLP) systems. The studies on bias in NLP are mainly delineated into two categories: bias in the embedding spaces, and bias in downstream tasks (Blodgett et al., 2020). For bias in downstream tasks, the analyses cover comprehensive topics, including machine translation (Stanovsky et al., 2019), language modeling (Bordia & Bowman, 2019), sentiment analysis (Kiritchenko & Mohammad, 2018) and toxicity detection (Dixon et al., 2018). The social bias in embedding spaces has been studied from two important perspectives: bias measurements and and debiasing methods. To measure the bias in an embedding space, Caliskan et al. (2017) proposed a Word Embedding Association Test (WEAT), which compares the similarity between two sets of target words and two sets of attribute words. May et al. (2019) further extended the WEAT to a Sentence Encoder Association Test (SEAT), which replaces the word embeddings by sentence embeddings encoded from pre-defined biased sentence templates. For debiasing methods, most of the prior works focus on word-level representations (Bolukbasi et al., 2016; Bordia & Bowman, 2019). The only sentence-level debiasing method is proposed by Liang et al. (2020), which learns bias directions by PCA and subtracts them in the embedding space.

heading: 4.2 CONTRASTIVE LEARNING

text: Contrastive learning is a broad class of training strategies that learns meaningful representations by making positive and negative embedding pairs more distinguishable. Usually, contrastive learning requires a pairwise embedding critic as a similarity/distance of data pairs. Then the learning objective is constructed by maximizing the margin between the critic values of positive data pairs and negative data pairs. Previously contrastive learning has shown encouraging performance in many tasks, including metric learning (Weinberger et al., 2006; Davis et al., 2007), word representation learning (Mikolov et al., 2013), graph learning (Tang et al., 2015; Grover & Leskovec, 2016), etc. Recently, contrastive learning has been applied to the unsupervised visual representation learning task, and significantly reduced the performance gap between supervised and unsupervised learning (He et al., 2020; Chen et al., 2020; Qian et al., 2020). Among these unsupervised methods, Chen et al. (2020) proposed a simple multi-view contrastive learning framework (SimCLR). For each image data, SimCLR generates two augmented images, and then the mutual information of the two augmentation embeddings is maximized within a batch of training data.

heading: 5 EXPERIMENTS

text: We first describe the experimental setup in detail, including the pretrained encoders, the training of FairFil, and the downstream tasks. The results of our FairFil are reported and analyzed, along with the previous Sent-Debias method. In general, we evaluate our neural debiasing method from two perspectives: (1) fairness: we compare the bias degree of the original and debiased sentence embeddings for debiasing performance; and (2) representativeness: we apply the debiased embeddings into downstream tasks, and compare the performance with original embeddings.

heading: 5.1 BIAS EVALUATION METRIC

text:
To evaluate the bias in sentence embeddings, we use the Sentence Encoder Association Test (SEAT) (May et al., 2019), which is an extension of the Word Embedding Association Test (WEAT) (Caliskan et al., 2017). The WEAT test measures the bias in word embeddings by comparing the distances of two sets of target words to two sets of attribute words. More specifically, denote X and Y as two sets of target word embeddings (e.g., X includes “male” words such as “boy” and “man”; Y contains “female” words like “girl” and “woman”). The attribute setsA and B are selected from some social concepts that should be “equal” to X and Y (e.g., career or personality words). Then the bias degree w.r.t attributes (A,B) of each word embedding t is defined as:
s(t,A,B) = meana∈A cos(t,a)−meanb∈B cos(t, b), (6)
where cos(·, ·) is the cosine similarity. Based on (6), the normalized WEAT effect size is:
dWEAT = meanx∈X s(x,A,B)−meany∈Ys(y,A,B)
stdt∈X∪Ys(t,A,B) . (7)
The SEAT test extends WEAT by replacing the word embeddings with sentence embeddings. Both target words and attribute words are converted into sentences with several semantically bleached sentence templates (e.g., “This is <word>”). Then the SEAT statistic is similarly calculated with (7) based on the embeddings of converted sentences. The closer the effect size is to zero, the more fair the embeddings are. Therefore, we report the absolute effect size as the bias measure.

heading: 5.2 PRETRAINED ENCODERS

text:
We test our neural debiasing method on BERT (Devlin et al., 2019). Since the pretrained BERT requires the additional fine-tuning process for downstream tasks, we report the performance of our FairFil under two scenarios: (1) pretrained BERT: we directly learn our FairFil network based on pretrained BERT without any additional fine-tuning; and (2) BERT post tasks: we fix the parameters of the FairFil network learned on pretrained BERT, and then fine-tune the BERT+FairFil together on task-specific data. Note that when fine-tuning, our FairFil will no longer update, which satisfies a fair comparison to Sent-Debias (Liang et al., 2020).
For the downstream tasks of BERT, we follow the setup from Sent-Debias (Liang et al., 2020) and conduct experiments on the following three downstream tasks: (1) SST-2: A sentiment classification task on the Stanford Sentiment Treebank (SST-2) dataset (Socher et al., 2013), on which sentence embeddings are used to predict the corresponding sentiment labels; (2) CoLA: Another sentiment classification task on the Corpus of Linguistic Acceptability (CoLA) grammatical acceptability judgment (Warstadt et al., 2019); and (3) QNLI: A binary question answering task on the Question Natural Language Inference (QNLI) dataset (Wang et al., 2018).

heading: 5.3 TRAINING OF FAIRFIL

text: We parameterize the fair filter network with one-layer fully-connected neural networks with the ReLU activation function. The score function g in the InfoNCE estimator is set to a two-layer fully-connected network with one-dimensional output. The variational approximation qθ in CLUB estimator is parameterized by a multi-variate Gaussian distribution qθ(w|d) = N(µ(d),σ2(d)), where µ(·) and σ(·) are also two-layer fully-connected neural nets. The batch size is set to 128. The learning rate is 1× 10−5. We train the fair filter for 10 epochs. For an appropriate comparison, we follow the setup of Sent-Debias (Liang et al., 2020) and select the same training data for the training of FairFil. The training corpora consist 183,060 sentences from the following five datasets: WikiText-2 (Merity et al., 201y), Stanford Sentiment Treebank (Socher et al., 2013), Reddit (V”olske et al., 2017), MELD (Poria et al., 2019) and POM (Park et al., 2014). Following Liang et al. (2020), we mainly select “gender” as the sensitive topic T , and use the same pre-defined word sets of sensitive attribute words and their replaceable words as Sent-Debias did. The word embeddings for training the debiasing regularizer is selected from the token embedding of the pretrained BERT.

heading: 5.4 DEBIASING RESULTS

text:
In Tables 2 and 3 we report the evaluation results of debiased embeddings on both the absolute SEAT effect size and the downstream classification accuracy. For the SEAT test, we follow the
setup in Liang et al. (2020), and test the sentence templates of Terms/Names under different domains designed by Caliskan et al. (2017). The column name Origin refers to the original BERT results, and Sent-D is short for Sent-Debias (Liang et al., 2020). FairFil− and FairFil (as FairF− and FairF in the tables) are our method without/with the debiasing regularizer in Section 3.3. The best results of effect size (the lower the better) and classification accuracy (the higher the better) are bold among Sent-D, FairFil−, and FairFil. Since the pretrained BERT does not correspond to any downstream task, the classification accuracy is not reported for it.
From the SEAT test results, our contrastive learning framework effectively reduces the gender bias for both pretrained BERT and fine-tuned BERT under most test scenarios. Comparing with Sent-Debias, our FairFil reaches a lower bias degree on the majority of the individual SEAT tests. Considering the average of absolute effect size, our FairFil is distinguished by a significant margin to Sent-Debias. Moreover, our FairFil achieves higher downstream classification accuracy than Sent-Debias, which indicates learning neural filter networks can preserve more semantic meaning than subtracting bias directions learned from PCA.
For the ablation study, we also report the results of FairFil without the debiasing regularizer, as in FairF−. Only with the contrastive learning framework, FairF− already reduces the bias effectively and even achieves better effect size than the FairF on some of the SEAT tests. With the debiasing regularizer, FairF has better average SEAT effect sizes but slightly loses in terms of the downstream performance. However, the overall performance of FairF and FairF− shows a trade-off between fairness and representativeness of the filter network.
We also compare the debiasing performance on a broader class of baselines, including word-level debiasing methods, and report the average absolute SEAT effect size on the pretrained BERT encoder. Both FairF− and FairF achieve a lower bias degree than other baselines. The word-level debiasing methods (FastText (Bojanowski et al., 2017) and BERT word (Bolukbasi et al., 2016))
have the worst debiasing performance, which validates our observation that the word-level debiasing methods cannot reduce sentence-level social bias in NLP models.

heading: 5.5 ANALYSIS

text:
To test the influence of data proportion on the model’s debiasing performance, we select WikiText-2 with 13,750 sentences as the training corpora following the setup in Liang et al. (2020). Then we randomly divide the training data into 5 equal-sized partitions. We evaluate the bias degree of the sentence debiasing methods on different combinations of the partitions, specifically with training data proportions (20%, 40%, 60%, 80%, 100%). Under each data proportion, we repeat the training 5 times to obtain the mean and variance of the absolute SEAT effect size. In Figure 2, we plot the bias degree of BERT post tasks with different training data proportions. In general, both Sent-Debias and FairFil achieve better performance and smaller variance when the proportion of training data is larger. Under a 20% training proportion, our FairFil can better remove bias in text encoder, which shows FairFil has better data efficiency with the contrastive learning framework.
To further study output debiased sentence embedding, we visualize the relative distances of attributes and targets of SEAT before/after our debiasing process. We choose the target words as “he” and “she.” Attributes are selected from different social domains. We first contextualize the selected words into sentence templates as described in Section 5.1. We then average the original/debiased embeddings of these sentence template and plot the t-SNE (Maaten & Hinton, 2008) in Figure 3. From the t-SNE, the debiased encoder provides more balanced distances from gender targets “he/she” to the attribute concepts.

heading: 6 CONCLUSIONS

text: This paper has developed a novel debiasing method for large-scale pretrained text encoder neural networks. We proposed a fair filter (FairFil) network, which takes the original sentence embeddings as input and outputs the debiased sentence embeddings. To train the fair filter, we constructed a multi-view contrast learning framework, which maximizes the mutual information between each sentence and its augmentation. The augmented sentence is generated by replacing sensitive words in the original sentence with words in a similar semantic but different bias directions. Further, we designed a debiasing regularizer that minimizes the mutual information between the debiased embeddings and the corresponding sensitive words in sentences. Experimental results demonstrate the proposed FairFil not only reduces the bias in sentence embedding space, but also maintains the semantic meaning of the embeddings. This post hoc method does not require access to the training corpora, or any retraining process of the pretrained text encoder, which enhances its applicability.

heading: ACKNOWLEDGEMENTS

text: This research was supported in part by the DOE, NSF and ONR.

title: FAIRFIL: CONTRASTIVE NEURAL DEBIASING METHOD FOR PRETRAINED TEXT ENCODERS