summaries: **(motivation)** Many approaches exist to generate counterfactual images, but what should be done when no training data is available? Current approaches lead to adversarial-looking approaches, which are not satisfying enough as an explanation.  **(approach)** The authors use Deep Inversion and a novel objective that ensures the data stays in distribution and propose a method to evaluate the quality of counterfactuals.  **(experiments)** The authors run experiments on Celebi and ISIC.   Summary: The paper proposed a novel approach to achieve deep model inversion for a pre-trained deep classifier using the classifier alone, without having access to the training dataset or generative models. Given a query image, the proposed deep model inversion generates a counterfactual explanation that is realistic looking and produces the desired classification outcome.    Earlier work on image synthesis by inverting deep classifiers either creates unrealistic images or creates perturbation of query image that successfully changes the classification decision but resulting pixel-level manipulations are not interpretable/meaningful (similar to an adversarial attack).   The proposed approach has four essential components: 1.	Counterfactual image is synthesized using an image prior. The authors experimented with an un-trained UNET-based model as a deep image prior (DIP) and Fourier mapping with SIREN activation as implicit neural representation (INR) for mapping noise to the counterfactual image. 2.	To ensure high similarity between counterfactual and query images, the authors used ISO to minimize discrepancies over the entire image and LSO to minimize differences between embeddings extracted from different layers of the deep classifier. 3.	To ensure that the generated counterfactual image is realistic and lies on the data manifold, the proposed model uses a manifold consistency (MC) loss. The MC-loss is quantified using an auxiliary loss predictor function in DEP or the radial basis kernel similarity in DUQ. Both methods provide ways to capture epistemic uncertainty due to out-of-distribution (OOD) samples. 4.	The model uses a functional consistency loss to ensure the counterfactual image produces a desired outcome from the deep classifier. Finally, the model uses progressive optimization to learn the image prior layer-by-layer.   The paper proposes a deep inversion approach that uses a pre-trained classifier to generate the counterfactual explanations of a given image. In other words, the method introduces small, discernible perturbations in an image that changes its class while keeping it realistic. The method makes use of certain inductive biases such as strong image priors, manifold consistency objective, and progressive optimization strategy to generate those counterfactual explanations. In this paper authors propose to generate counterfactuals from a query image and a given trained deep classifier. One is interested to find out a semantically reasonable alternative explanation, where image would be classified from class y to y'. This is all performed without the help of an explicit generator (such as GAN). 
abstractText: Explanation techniques that synthesize small, interpretable changes to a given image while producing desired changes in the model prediction have become popular for introspecting black-box models. Commonly referred to as counterfactuals, the synthesized explanations are required to contain discernible changes (for easy interpretability) while also being realistic (consistency to the data manifold). In this paper, we focus on the case where we have access only to the trained deep classifier and not the actual training data. While the problem of inverting deep models to synthesize images from the training distribution has been explored, our goal is to develop a deep inversion approach to generate counterfactual explanations for a given query image. Despite their effectiveness in conditional image synthesis, we show that existing deep inversion methods are insufficient for producing meaningful counterfactuals. We propose DISC (Deep Inversion for Synthesizing Counterfactuals) that improves upon deep inversion by utilizing (a) stronger image priors, (b) incorporating a novel manifold consistency objective and (c) adopting a progressive optimization strategy. We find that, in addition to producing visually meaningful explanations, the counterfactuals from DISC are effective at learning classifier decision boundaries and are robust to unknown test-time corruptions.
authors.name: Jayaraman J. Thiagarajan
authors.name: Lawrence Livermore
authors.name: Vivek Narayanaswamy
authors.name: Deepta Rajan
authors.name: Akshay Chaudhari
references.authors: S. Verma J. Dickerson K. Hines
references.title: Counterfactual explanations for machine learning: A review
references.venue: arXiv preprint arXiv:2010.10596
references.authors: RR. Selvaraju M. Cogswell A. Das R. Vedantam D. Parikh D. Batra
references.title: Grad-cam: Visual explanations from deep networks via gradient-based localization
references.venue: Proceedings of the IEEE international conference on computer vision, pages 618–626
references.authors: H. Lakkaraju N. Arsov O. Bastani
references.title: Robust black box explanations under distribution shift
references.venue: International Conference on Machine Learning (ICML)
references.authors: A. Shrikumar P. Greenside A. Kundaje
references.title: Learning important features through propagating activation differences
references.venue: International Conference on Machine Learning, pages 3145–3153. PMLR
references.authors: MT Ribeiro S. Singh C. Guestrin
references.title: Why should i trust you?" explaining the predictions of any classifier
references.venue: Proc. of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135–1144
references.authors: MT. Ribeiro S. Singh C. Guestrin
references.title: Anchors: High-precision model-agnostic explanations
references.venue: Proceedings of the AAAI Conference on Artificial Intelligence, volume 32
references.authors: AV. Looveren J. Klaise
references.title: Interpretable counterfactual explanations guided by prototypes
references.venue: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 650–665. Springer
references.authors: A. Dhurandhar P. Chen R. Luss C. Tu P. Ting K. Shanmugam P. Das
references.title: Explanations based on the missing: Towards contrastive explanations with pertinent negatives
references.venue: Advances in Neural Information Processing Systems, volume 31
references.authors: S. Sumedha P. Brian C. Junxiang B. Kayhan
references.title: Explanation by progressive exaggeration
references.venue: International Conference on Learning Representations
references.authors: Y. Goyal Z. Wu J. Ernst D. Batra D. Parikh S. Lee
references.title: Counterfactual visual explanations
references.venue: International Conference on Machine Learning, pages 2376–2384. PMLR
references.authors: A. Mordvintsev C. Olah
references.title: and M
references.venue: Tyka. Inceptionism: Going deeper into neural networks
references.authors: A. Mahendran A. Vedaldi
references.title: Visualizing deep convolutional neural networks using natural pre-images
references.venue: International Journal of Computer Vision, 120(3):233–255
references.authors: H. Yin P. Molchanov JM. Alvarez Z. Li A. Mallya D. Hoiem NK. Jha J. Kautz
references.title: Dreaming to distill: Data-free knowledge transfer via deepinversion
references.venue: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8715–8724
references.authors: S. Ioffe C. Szegedy
references.title: Batch normalization: Accelerating deep network training by reducing internal covariate shift
references.venue: International conference on machine learning, pages 448–456. PMLR
references.authors: Geoffrey H V. Oriol D. Jeffrey
references.title: Distilling the knowledge in a neural network
references.venue: In NIPS Deep Learning and Representation Learning Workshop,
references.authors: D. Ulyanov A. Vedaldi V. Lempitsky
references.title: Deep image prior
references.venue: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9446–9454
references.authors: V. Sitzmann J. Martel A. Bergman D. Lindell G. Wetzstein
references.title: Implicit neural representations with periodic activation functions
references.venue: Advances in Neural Information Processing Systems, 33
references.authors: D. Yoo I.S Kweon
references.title: Learning loss for active learning
references.venue: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 93–102
references.authors: JJ. Thiagarajan V. Narayanaswamy R. Anirudh PT. Bremer A. Spanias
references.title: Accurate and robust feature importance estimation under distribution shifts
references.venue: Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7891–7898
references.authors: J. Van Amersfoort L. Smith YW. Teh Y. Gal
references.title: Uncertainty estimation using a single deep deterministic neural network
references.venue: International Conference on Machine Learning, pages 9690–9700. PMLR
references.authors: A. Dosovitskiy T. Brox
references.title: Inverting visual representations with convolutional networks
references.venue: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4829–4837
references.authors: A. Mahendran A. Vedaldi
references.title: Understanding deep image representations by inverting them
references.venue: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5188–5196
references.authors: D. Nemirovsky N. Thiebaut Y. Xu A. Gupta
references.title: Countergan: Generating realistic counterfactuals with residual generative adversarial nets
references.venue: arXiv preprint arXiv:2009.05199
references.authors: A. Sauer A. Geiger
references.title: Counterfactual generative networks
references.venue: ICLR
references.authors: O. Ronneberger P. Fischer T. Brox
references.title: U-net: Convolutional networks for biomedical image segmentation
references.venue: International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer
references.authors: Y. Chen S. Liu X. Wang
references.title: Learning continuous image representation with local implicit image function
references.venue: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8628–8638
references.authors: M. Tancik PP. Srinivasan B. Mildenhall S. Fridovich-Keil N. Raghavan U. Singhal R. Ramamoorthi JT. Barron R. Ng
references.title: Fourier features let networks learn high frequency functions in low dimensional domains
references.venue: NeurIPS
references.authors: Y. LeCun L. Bottou Y. Bengio P. Haffner
references.title: Gradient-based learning applied to document recognition
references.venue: Proceedings of the IEEE, 86(11):2278–2324
references.authors: G. Daras J. Dean A. Jalal AG. Dimakis
references.title: Intermediate layer optimization for inverse problems using deep generative models
references.venue: arXiv preprint arXiv:2102.07364
references.authors: T. Karras S. Laine M. Aittala J. Hellsten J. Lehtinen T. Aila
references.title: Analyzing and improving the image quality of stylegan
references.venue: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110–8119
references.authors: Z. Liu P. Luo X. Wang X. Tang
references.title: Deep learning face attributes in the wild
references.venue: In Proceedings of International Conference on Computer Vision (ICCV),
references.authors: N. Codella V. Rotemberg P. Tschandl M.E. Celebi S. Dusza D. Gutman B. Helba A. Kalloo K. Liopyris M. Marchetti
references.title: et al
references.venue: Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902.03368
references.authors: P. Tschandl C. Rosendahl H. Kittler
references.title: The ham10000 dataset
references.venue: a large collection of multisource dermatoscopic images of common pigmented skin lesions. Scientific data, 5:180161
references.authors: K. He X. Zhang S. Ren J. Sun
references.title: Deep residual learning for image recognition
references.venue: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778
references.authors: DP. Kingma J. Ba
references.title: Adam: A method for stochastic optimization
references.venue: International Conference on Learning Representations, San Diego, California
references.authors: C. Chang E. Creager A. Goldenberg D. Duvenaud
references.title: Explaining image classifiers by counterfactual generation
references.venue: International Conference on Learning Representations
sections.heading: 1 Introduction
sections.text: With the growing need for deploying deep black-box models into critical decision-making, there is an increased emphasis on explainability methods that can reveal intricate relationships between data signatures (e.g., image features) and predictions. In this context, the so-called counterfactual (CF) explanations [1] that synthesize small, interpretable changes to a given image while producing desired changes in model predictions to support user-specified hypotheses (e.g., progressive change in predictions) have become popular. Though counterfactual explanations provide more flexibility over conventional techniques, such as feature importance estimation [2, 3, 4, 5, 6], by exploring the vicinity of a query image, an important requirement to produce meaningful counterfactuals is to produce discernible local perturbations (for easy interpretability) while being realistic (close to the underlying data manifold). Consequently, existing approaches rely extensively on pre-trained generative models to synthesize plausible counterfactuals [1, 7, 8, 9, 10]. By design, this ultimately restricts their utility to scenarios where one cannot access the training data or pre-trained generative models, for example, due to privacy requirements commonly encountered in many practical applications.
  In this paper, we focus on the problem where we have access only to trained deep classifiers and not the actual training data or generative models. Synthesizing images from the underlying data distribution
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  by inverting a deep model, while not requiring access to training data, is a well investigated topic of research. For e.g., Deep Dream [11] synthesizes class-conditioned images by manipulating a noisy image directly in the space of pixels (or more formally Image Space Optimization (ISO)) constrained by image priors such as total variation [12] to regularize this ill-posed inversion. However, Deep Dream is known to produce images that look unrealistic, often very different from the training images, thus limiting their use in practice. Consequently, Yin et al., proposed DeepInversion [13] that performs image synthesis in the latent space of a pre-trained classifier (Latent Space Optimization (LSO)) and leverages layer-specific statistics (from batchnorm [14]) to constrain the images to be consistent with the training data distribution. This was showed to produce higher-quality images, particularly in the context of performing knowledge distillation [15] using the synthesized images.
  Proposed Work. In contrast, this work aims to develop a deep model inversion approach that generates counterfactual explanations by exploring the vicinity of a given query image, instead of synthesizing an arbitrary realization from the entire image distribution. As illustrated in the example in Figure 1, existing deep inversion methods are ineffective when natively adopted for counterfactual generation. Due to use of weak priors, and the severely ill-posed nature of the problem, it introduces irrelevant pixel manipulations that easily satisfy the desired change in prediction. Hence, we propose DISC (Deep Inversion for Synthesizing Counterfactuals) that improves upon conventional deep model inversion by utilizing: (i) stronger image priors through the use of deep image priors [16]
  (DIP) and implicit neural representations [17] (INR); (ii) a novel manifold consistency objective that ensures the counterfactual remains close to the underlying manifold; and (iii) a progressive optimization strategy to effectively introduce discernible, yet meaningful, changes to the query image.
  From Figure 1, we find that our approach produces meaningful image manipulations, in order to change the prediction to the smiling class, while other deep inversion strategies cannot. Using empirical studies, we show that DISC consistently produces visually meaningful explanations, and that the counterfactuals from DISC are effective at learning model decision boundaries and are robust to unknown test-time corruptions.
  Our Contributions.
  1. A general framework to produce counterfactuals on-the-fly using deep model inversion;
  2. Novel objectives to ensure consistency to the data manifold. We explore two different strategies based on direct error prediction [18, 19] and deterministic uncertainty estimation [20];
  3. A progressive optimization strategy to introduce discernible changes to a given query image, while satisfying the manifold consistency requirement;
  4. A classifier discrepancy metric to evaluate the quality of counterfactuals;
  5. Empirical studies using natural image and medical image classifiers to demonstrate the effectiveness of DISC over a variety of baselines and ablations.
sections.heading: 2 Related Work
sections.text: Image Synthesis from Classifiers. Inverting a pre-trained deep model is a popular strategy for image synthesis in scenarios where there is no access to the underlying training data. While deep model inversion-based methods such as Deep Dream [11] and DeepInversion [13] have been successful in generating class-conditioned images, there have been other extensions to such approaches. For example, Dosovitskiy et al. [21] proposed to invert representations of a pre-trained CNN to obtain insights about what a deep classifier has learned. On similar lines, Mahendran et al. [12, 22] addressed the problem of pre-image recovery, which in essence attempts to recover an arbitrarily
  encoded representation (in the latent space of a classifier) to a realization on the (unknown) image manifold and to enable model diagnosis. Ulyanov et al. [16] improved upon this ill-posed inversion by utilizing a strong image prior in the form of deep image priors (DIP). They also explored the related problem of activation-maximization [16], where the goal is to generate an image that maximizes the activation of a given output neuron in a pre-trained classifier, and demonstrated the effectiveness of DIP. Despite the effectiveness of the deep model inversion methods for image synthesis, we find that such methods cannot be natively adopted for CF generation and are insufficient for producing meaningful pixel manipulations to a given query image.
  Counterfactual Generation. Existing methods extensively rely on generative models to provide counterfactuals that explain the decisions of a black-box model. Examples including CounteRGAN [23], Counterfactual Generative Networks (CGNs) [24] and the methods reported in [7, 8], have clearly demonstrated the use of generative models in synthesizing high-quality CFs for any user-specified hypothesis on the predictions. However, this requirement of access to training data or generative models can be infeasible in practical scenarios, for e.g., restrictions arising due to privacy requirements. In contrast, our approach formulates the problem of counterfactual generation using deep model inversion, and can produce meaningful counterfactuals using only the trained classifier.
sections.heading: 3 Proposed Approach
sections.text: In this section, we describe our approach for counterfactual generation that improves upon deep model inversion, and introduce a new classifier discrepancy metric for evaluating CFs. There are four key components that are critical to designing classifier-based counterfactual generators: (i) choice of metric for semantics preservation; (ii) choice of image priors to regularize image synthesis; (iii) manifold consistency to ensure that the synthesized counterfactual lies close to the true data manifold; and (iv) progressive optimization strategy to introduce gradual meaningful changes to a query image. Figures 2a and 2b provide overview of DISC implemented with deep image prior and INR-based for regularizing the counterfactual optimization process.
  In its simplest form, for a given query x, a counterfactual explanation can be obtained as follows: arg min
  x̄ d(x̄, x) s.t. F(x̄) = ȳ, x̄ ∈M(x), (1)
  where x̄ = C(x) is a counterfactual explanation for x, F is a pre-trained classifier model,M denotes the data manifold and ȳ = y + δ is the desired change in the prediction. The metric d(., .) measures the discrepancy between the query image and the counterfactual (i.e., semantics preservation).
sections.heading: 3.1 Choice of Metric for Semantics Preservation
sections.text: We can measure the discrepancy between a query and its CF explanation, d(x̄, x), in the pixel space or in the latent space of the classifier. We now describe the high-level formulation of deep model inversion-based CF generation.
  a. Image Space Optimization (ISO). ISO for counterfactual generation involves the ill-posed optimization of an input x̄ directly in the pixel space to generate an image semantically similar to the query x, while being consistent with a user-hypothesis on the prediction, ȳ. Strategies such as Deep Dream [11] perform ISO to synthesize artistic variations of images. Mathematically,
  arg min x̄ d(x̄, x) +R(x̄) s.t. F(x̄) = ȳ. (2)
  whereR(x̄) is a suitable image prior to regularize the optimization.
  b. Latent Space Optimization (LSO). LSO refers to the ill-posed problem of inverting an arbitrarily encoded representation from the latent space of a deep classifier to a realization on the (unknown) image manifold. Let Ψl(.) denote the lth differentiable layer of the deep classifier. Then, counterfactual generation using LSO can be mathematically formulated as
  arg min x̄ ∑ l d(Ψl(x̄),Ψl(x)) +R(x̄) s.t. F(x̄) = ȳ. (3)
  Approaches such as DeepInversion [13] perform LSO for conditional image synthesis (though with distribution-level comparison instead of our sample-level comparison) and achieve visually superior images when compared to ISO approaches.
sections.heading: 3.2 Choice of Image Priors
sections.text: As observed from (2) and (3), the choice of the regularizer or image prior R(.) is central towards regularizing and tractably solve this challenging inverse problem. A variety of image priors have been proposed in the literature, and we investigate the following in this work.
  a. Total Variation + `2. Total variation [22] (TV) is a popular regularizer that encourages images to contain piece-wise constant patches while the `2 norm regularizes the range and energy of the image to remain within a given interval. The TV norm and the `2 norm are given by:
  RTV (x̄) = ∑ i,j √ (x̄i,j+1 − x̄i,j)2 + (x̄i+1,j − x̄i,j)2; R`2(x̄) = √∑ i,j ‖x̄i,j‖2 (4)
  b. Deep Image Priors (DIP). A Deep Image Prior (DIP) [16] leverages the structure of an untrained, carefully tailored convolutional neural network (e.g., U-Net [25]) to generate images and solve a variety of ill-posed restoration tasks in computer vision. DIP has been found to produce high-quality reconstructions, based on the key insight that the structure of the network itself can act as a regularizer. Consequently, the synthesized image is re-parameterized in terms of the weights θ of the prior model fθ i.e x̄ = fθ(z).
  c. Implicit Neural Representations (INR). We also considered an alternative approach based on INR, which provide a cheap and convenient way to learn a continuous mapping from the image coordinates to the pixel values (RGB). While they have been found to be effective for image/volume rendering and designing generative models [26], we explore their use in deep model inversion (details in appendix). We build upon two key results to design our INR-based counterfactual generators:
  (i) Fourier mapping: Based on NTK (neural tangent kernel) theory, [27] showed that using Fourier mapping can recover high-frequency features in low-dimensional coordinate-based image reconstruction. Hence, we use a Fourier feature mapping z to featurize 2−D input coordinates v ∈ [0, 1]2 before passing them through a coordinate-based MLP:
  z(v) = [a1 cos(2πb T 1 v), a1 sin(2πb T 1 v), · · · ].
  Using a set of randomly chosen sinusoids, this maps the input points to the surface of a highdimensional hypersphere. Training the MLP network on these embedded points corresponds to kernel regression with the stationary composed NTK hNTK ◦ hz, where hNTK denotes the neural tangent kernel corresponding to the MLP;
  (ii) SIREN activation: In [17], Sitzmann et al. showed that periodic activation functions are better suited for recovering natural images and their derivatives, when compared to standard activation functions. More specifically, SIREN uses a sinusoid activation Φ(x) = sin(Wx + b). We find that using both a Fourier mapping coupled with SIREN activation leads to a very strong image prior.
sections.heading: 3.3 Manifold Consistency
sections.text: A key constraint in (1) that is not included in the formulations in (2), (3) is the manifold consistency, and interestingly, this is not inherently satisfied in deep model inversion. Consequently, even with a strong image prior, it can synthesize images that do not belong to the original data distribution. This
  can be particularly challenging when producing CFs that represent change in class labels, wherein one expects patterns specific to a target class to be emphasized. To address this challenge, we extend the formulation in (3) (and equivalently (2)) to include a manifold consistency constraint, that is defined directly based on the classifier, without assuming access to class-specific statistics in the latent space. More specifically,:
  arg min x̄ λ1 ∑ l d(Ψl(x̄),Ψl(x)) + λ2Lmc(x̄;F) + λ3Lfc(F(x̄), ȳ). (5)
  The first term for semantics preservation is same as that of (3) and is used to ensure that the inherent semantics of the query image is retained in the generated counterfactual (implemented as the `2 error). The second term Lmc (manifold consistency) penalizes solutions that do not lie close to the data manifold and is designed by assuming access only to the classifier F . The final loss term Lfc (functional consistency) ensures that the prediction for the counterfactual matches the desired target ȳ, e.g., categorical cross entropy. As illustrated in Figure 3, the manifold consistency objective plays a central role in deep inversion-based CF generation. Even with a strong image prior (DIP in this example) and LSO, the generator produces out-of-distribution (OOD) CFs (with missing pixels) as guided by the semantic preservation term. Though the synthesized CF produces the desired class label with the classifier F , the explanation is not interpretable. In contrast, including the Lmc term (using DEP explained next) leads to a meaningful counterfactual that automatically fills in the missing pixels. Note that, this example is different from prior-based image inpainting [16], where a known
  mask is used to alter the loss function to recover the missing pixels. In this work, we explore two different strategies to implement Lmc: a. Direct Error Prediction (DEP). Recently, in [19], it was found that a loss predictor trained jointly with the classifier can be used to effectively detect distribution shifts and obtain accurate uncertainty estimates for a given sample. Assuming that, a classifier model F is trained to optimize the primary loss Lpri = LCE(F(x), y), we construct an auxiliary loss predictor G to estimate the loss s = G(x) = Lpri. Similar to [18, 19], we utilize an auxiliary loss function Laux(s, ŝ) to train the parameters of G. In particular, we adopt the contrastive loss which aims to preserve the ordering of samples based on their corresponding losses from F . Let si and sj denote the
  losses of samples xi and xj , while the corresponding estimates from G are ŝi and ŝj respectively. Now,
  Laux = ∑ (i,j) max ( 0,−I(si, sj).(ŝi − ŝj) + γ ) , (6)
  where I(si, sj) = {
  1, if si > sj , −1, otherwise.
  Here γ is an optional margin parameter (γ = 1 for our implementation). The overall objective for the joint optimization of the classifier and loss predictor is given by
  Ltotal = β1Lpri + β2Laux. (7)
  After the models are trained, we implement the manifold consistency term Lmc directly based on the loss predictor, since the losses indicate regimes where the model fails to make an accurate prediction:
  Lmc = ‖G(x̄)− s∗‖1, (8) where s∗ denotes the target loss to be achieved. In our experiments, we set s∗ as the median error estimate from DEP on a held-out validation set.
  b. Deterministic Uncertainty Quantification (DUQ). The recently proposed DUQ [20] method is based on Radial Basis Function (RBF) [28] networks and has been showed to be highly effective at OOD detection. In its formulation, a model is comprised of a deep feature extractor F , an exponential
  kernel function along with a set of prototypical feature vectors (or centroids) for each class. DUQ is trained by optimizing the kernel distance between the features from F and the class-specific centroids and using a moving average process to update the centroids. Once DUQ is trained, the uncertainty can be measured as the distance between the model output and the closest centroid. Details regarding DUQ training can be found in the appendix. In this work, we implement Lmc using a margin-based loss that maximizes the kernel similarity of the synthesized CF with the centroid of the target class, relative to the source class. Denoting the kernel similarity for a CF x̄ with the centroid for class y as K(F(x̄), φ(y)), where φ(y) corresponds the pre-computed centroid for class y, we define:
  Lmc = max ( K(F(x̄), φ(y))−K(F(x̄), φ(ȳ)) + τ, 0 ) , (9)
  which indicates that kernel similarity w.r.t. the target class ȳ should be greater than that with the source class y at least by the margin τ (set to 0.5 in our experiments).
sections.heading: 3.4 Progressive Optimization
sections.text: CF generation is a highly under-constrained problem, that even with a strong image prior and the proposed manifold consistency constraint, it can easily converge to trivial solutions, i.e., irrelevant image manipulations. For example, one might expect to introduce large discernible changes by reducing the penalty λ1 for semantics preservation. However, given the large solution space (defined by the number of parameters in the DIP/INR generator fθ), this often leads to unrealistic images. To circumvent this, we propose to adopt a progressive optimization strategy that gradually increases the number of layers in fθ to be optimized and steadily relaxing the penalty λ1 (by factor κ) to allow for larger, yet interpretable, changes. More specifically, denoting the number of layers in fθ by L, in each iteration we train the parameters of the first i layers (i is incremented by 1 in the subsequent iteration) while keeping the parameters of the remaining L− i layers at their initial state (details on how the layers are chosen for DIP and INR based generators can be found in the appendix). A similar strategy has been shown to be effective for ill-posed restoration tasks using large-scale generative models such as Style-GAN [29, 30]. An outline of this progressive optimization process is provided in the appendix. We find that such a progressive optimization leads to significantly better quality solutions allowing meaningful traversal from one class to another.
sections.heading: 3.5 Evaluating Quality of CF Explanations using Classifier Discrepancy
sections.text: A desired property in query-based explainers is that the synthesized changes are both interpretable and representative of the target class (e.g., smiling). To systematically evaluate the latter property, we propose the following synthetic experiment: Given a binary classifier F and training images, i.e., X0, belonging to Class 0, we use our CF generator to synthesize examples for Class 1, i.e., X̄1 (using Class 0 images as input), and finally build a secondary classifier Fc using [X0, X̄1]. The quality of the counterfactuals can thus be measured using the gap between the performance of F and Fc on a common test set. We refer to this score as classifier discrepancy (CD).
sections.heading: 4 Experiment Setup
sections.text: Datasets. (i) CelebA Faces [31]: This dataset contains 202,599 images along with a wide-range of attributes. For our experiments, we consider 3 different attributes, namely smiling, bald and young. Note, we train a classifier for predicting each of the attributes independently. We report the results for bald and young attributes in the appendix; (ii) ISIC 2018 Skin Lesion Dataset [32]: This lesion diagnosis challenge dataset contains a total of 10, 015 dermoscopic lesion images from the HAM10000 database [33]. Each image is associated with one out of 7 disease states: Melanoma (MEL), Melanocytic nevus (MN), Basal cell carcinoma (BCC), Actinic keratosis (AK), Benign keratosis (BK), Dermatofibroma (DF) and Vascular lesion (VASC). Note, in all cases, we used a stratified 90− 10 data split to train the classifiers. Classifier Design. For all experiments, we resized the images to size 96× 96 and used the standard ResNet-18 architecture [34] to train the classifier model with the Adam optimizer [35], batch size 128, learning rate 1e− 4 and momentum 0.9. For the DEP implementation (Section 3.3), we performed average pooling on feature maps from each of the residual blocks in ResNet-18, and applied a linear layer of 128 units with ReLU activation. The hyper-parameters in (7) were set at β1 = 1.0 and
  β2 = 0.5. For the case of DUQ, we set both the length scale parameter and the gradient penalty to 0.5.
  Image Generator Design. For generator design, the deep image prior used the standard U-Net architecture and input noise images drawn from the uniform distribution U [−1, 1]. For INR, we chose 256 random sinusoids with frequencies bi drawn from a Gaussian distribution with mean 0 and variance 100 to compute the Fourier mapping for the input coordinates.
sections.heading: 5 Findings
sections.text: LSO Offers Better Feature Manipulation Control over ISO with Different Priors. An important design component in DISC is how we compute the semantic discrepancy between query x and CF x̄ - either in the pixel space using ISO or in the latent space of the classifier using LSO. We perform a comparative analysis of their behavior in CF generation using CelebA faces, in particular when manipulating an image from the non-smiling class to be classified as smiling, by varying the choice of image priors. As observed in Figure 4, though LSO produces higher quality images compared to ISO when using a weak image prior (TV + `2), that quality gap vanishes with the use of a stronger prior, e.g., DIP. However, in terms of producing discernible changes that reflect the properties of the smiling class, neither approach is sufficient. In particular, ISO shows a higher risk of making minimal, irrelevant modifications (refer to difference images |x− x̄| in Figure 4) that drive the prediction to a desired label and hence, similar to [13], we recommend the use of LSO but with stronger image priors to design CF generators.
  Manifold Consistency is Essential to Produce Meaningful Explanations. As showed in the previous experiment, deep model inversion does not produce discernible (and interpretable) image changes when applied for CF generation. In this context, we explore the impact of enforcing manifold consistency in DISC. In particular, we compare the following LSO-based CF generation implementations (with INR prior): (i) no manifold consistency; (ii) DUQ-based Lmc from (9); and (iii) DEP-based Lmc from (8). From the results in Figure 5, we find that the proposed DEP objective significantly improves over the standard LSO (with no Lmc) by introducing appropriate pixel manipulations near the mouth and cheeks in all examples and clearly represents the underlying semantics of the smiling class. In contrast, the RBF network-based DUQ performs very similar to the LSO + INR baseline and
  this emphasizes the inability of the kernel similarity metric to detect mild distribution shifts in data, though it has been proven successful for detecting severely OOD samples.
  INR based Priors Produce Highly Concentrated Changes over DIP. Although both DIP and INR are effective, a more rigorous comparison of those two image priors is required. For this purpose, we utilize two important evaluation metrics, namely: (i) ability to produce large discernible changes, measured using `2 error between x and x̄ in the pixel-space; and (ii) ability to produce concentrated image manipulations [36], measured by thresholding (< 0.05) the difference image |x − x̄| and determining the area of the largest bounding box that contains all the non-zero values (between 0 and 1). Ideally, explanations that have a larger MSE and consistently lower concentration are more likely to produce easily interpretable, localized changes. The results in Table 1 are obtained by randomly choosing 5000 images from the non-smiling class and synthesizing the corresponding counterfactuals for smiling. Similarly, for the ISIC2018 dataset, we used 800 randomly chosen images from the MEL class and generated CFs for NEV. The naïve LSO + TV + `2 baseline produces counterfactuals with a significantly large MSE as well as concentration score, indicating that the CFs are uninterpretable and contain irrelevant perturbations all over the image. As expected, incorporating a stronger prior improves the concentration significantly, while also being highly conservative in terms of MSE, i.e., non-discernible changes. Finally, enforcing manifold consistency using DEP achieves an optimal trade-off between the two metrics and produces meaningful CFs (Figures 6 and 7). In particular, INR based generators produce highly concentrated image manipulations.
  LSO + INR + DEP Produces CFs with Low Classifier Discrepancy Score. We now evaluate the quality of the counterfactuals using the proposed classifier discrepancy (CD) score. For this purpose, we consider a random subset of 5000 images each from non-smiling (X0) and smiling (X1) classes respectively. Following the strategy described in Section 3.5, we train the classifiers F and Fc, and measure the CD score as the difference in test accuracies, Acc.(F , Xtest)− Acc.(Fc, Xtest). Similarly, we repeat a simple evaluation for ISIC data by using 800 images from class MEL asX0 and images from class NEV as X1. From Table 1, we find that, using manifold consistency along with a strong image prior produces significantly lower CD scores (0.08 with LSO + INR + DEP on CelebA), when compared with LSO without manifold consistency (0.26 with LSO + INR). In particular, using INR + DEP fairs the best and consistently produces highly meaningful counterfactuals.
  DISC Explanations are Robust Under Test-Time Distribution Shifts. In several practical applications, we often encounter shifts between the train and test distributions which makes model deployment challenging. Hence, we evaluate the behavior of the proposed approach under unknown distribution shifts at test time. Note, we train the classifier on the clean CelebA faces dataset without introducing any corruptions. However, when we introduce corruptions at test-time, we find that (see Figure 8), our approach (LSO + INR + DEP) is able to robustly manipulate the appropriate regions in the query image. This can be attributed both to the ability of DEP to reflect challenging distribution shifts with higher error estimates [19] and our progressive optimization strategy to induce larger yet semantically meaningful changes (i.e., inherent noise clean up).
sections.heading: 6 Conclusions
sections.text: In this paper, we presented DISC, a general approach to design counterfactual generators for any deep classifier, without requiring access to the training data or generative models. We draw connections to the problem of deep model inversion and extend it to support counterfactual generation. DISC can learn a CF generator on-the-fly by leveraging different image priors and manifold consistency constraints, along with a progressive optimization strategy, to synthesize highly-plausible explanations. Future extensions to this work include exploring the use of multiple target attributes simultaneously in our optimization and applying this method to time-varying data.
sections.heading: 7 Acknowledgements
sections.text: This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.
title: Designing Counterfactual Generators using Deep Model Inversion