summaries: This paper provides an interesting pos-hoc explanation method to identify relevant features in an input that may inform a trained neural model's prediction. The task is to identify a binary mask over input image/text such that the masked input yields almost similar prediction as original input. The author formulates this as an SMT solver task, but instead of making sure that the output prediction is similar (which involve multiple time consuming pass over potentially huge networks), they make sure that high influential neurons in first layer of the network are still activated. This provides a less time consuming way to evaluate invariance of masked input.

summaries: This paper addresses the question of identifying which input features are most important for a neural network's decision. To do so, it frames the problem as an SMT problem that seeks to select the best input features, without changing the state of the first layer too much. The paper shows experiments, primarily on image classification, and also examples of how the approach may be applied to text classification.

summaries: This paper presents a method to encode the minimal input feature discovery problem -- finding the minimal set of features in a input that is necessary for a prediction -- into a form that can is amenable to satisfiability modulo theory (SMT) solvers.  In particular they first use the integrated gradients methods to score first-layer neurons on the degree to which they influence the prediction.  Then, they produce and solve an SMT problem that finds the minimal mask that changes these influential neurons.  They demonstrate their approach on several problems.

abstractText: Symbolic techniques based on Satisfiability Modulo Theory (SMT) solvers have been proposed for analyzing and verifying neural network properties, but their usage has been fairly limited owing to their poor scalability with larger networks. In this work, we propose a technique for combining gradient-based methods with symbolic techniques to scale such analyses and demonstrate its application for model explanation. In particular, we apply this technique to identify minimal regions in an input that are most relevant for a neural network’s prediction. Our approach uses gradient information (based on Integrated Gradients) to focus on a subset of neurons in the first layer, which allows our technique to scale to large networks. The corresponding SMT constraints encode the minimal input mask discovery problem such that after masking the input, the activations of the selected neurons are still above a threshold. After solving for the minimal masks, our approach scores the mask regions to generate a relative ordering of the features within the mask. This produces a saliency map which explains “where a model is looking” when making a prediction. We evaluate our technique on three datasets MNIST, ImageNet, and Beer Reviews, and demonstrate both quantitatively and qualitatively that the regions generated by our approach are sparser and achieve higher saliency scores compared to the gradient-based methods alone. Code and examples are at https://github.com/google-research/google-research/tree/master/smug_saliency

authors:
David Alvarez-Melis
Tommi S Jaakkola

title: On the robustness of interpretability methods

venue: arXiv preprint arXiv:1806.08049,

authors:
Sebastian Bach
Alexander Binder
Grégoire Montavon
Frederick Klauschen
Klaus-Robert Müller
Wojciech Samek

title: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation

venue: PloS one,

authors:
Clark Barrett
Cesare Tinelli

title: Satisfiability modulo theories

venue: In Handbook of Model Checking,

authors:
Nikolaj Bjørner
Anh-Dung Phan
Lars Fleckenstein

title: νz-an optimizing smt solver

venue: In International Conference on Tools and Algorithms for the Construction and Analysis of Systems,

authors:
Brandon Carter
Jonas Mueller
Siddhartha Jain
David Gifford

title: What made you do this? understanding black-box decisions with sufficient input subsets

venue: arXiv preprint arXiv:1810.03805,

authors:
Piotr Dabkowski
Yarin Gal

title: Real time image saliency for black box classifiers

venue: In Advances in Neural Information Processing Systems,

authors:
Jia Deng
Wei Dong
Richard Socher
Li-Jia Li
Kai Li
Li Fei-Fei

title: Imagenet: A large-scale hierarchical image database

authors:
Ruth C Fong
Andrea Vedaldi

title: Interpretable explanations of black boxes by meaningful perturbation

venue: In Proceedings of the IEEE International Conference on Computer Vision,

authors:
D. Gopinath
H. Converse
C. Pasareanu
A. Taly

title: Property inference for deep neural networks

venue: In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE),

authors:
Divya Gopinath
Hayes Converse
Corina Pasareanu
Ankur Taly

authors:
Sara Hooker
Dumitru Erhan
Pieter-Jan Kindermans
Been Kim

title: A benchmark for interpretability methods in deep neural networks

authors:
Xiaowei Huang
Marta Kwiatkowska
Sen Wang
Min Wu

title: Safety verification of deep neural networks

venue: In International Conference on Computer Aided Verification,

authors:
Alexey Ignatiev
Nina Narodytska
Joao Marques-Silva

title: Abduction-based explanations for machine learning models

venue: In Proceedings of the AAAI Conference on Artificial Intelligence,

authors:
Andrei Kapishnikov
Tolga Bolukbasi
Fernanda Viégas
Michael Terry

title: Xrai: Better attributions through regions

authors:
Guy Katz
Clark Barrett
David Dill
Kyle Julian
Mykel Kochenderfer

title: Reluplex: An efficient smt solver for verifying deep neural networks, 2017

authors:
Yann LeCun
Corinna Cortes
CJ Burges

title: Mnist handwritten digit database

authors:
Jan Macdonald
Stephan Wäldchen
Sascha Hauch
Gitta Kutyniok

title: A rate-distortion framework for explaining neural network decisions

venue: arXiv preprint arXiv:1905.11092,

authors:
Julian McAuley
Jure Leskovec
Dan Jurafsky

title: Learning attitudes and attributes from multi-aspect reviews

venue: In 2012 IEEE 12th International Conference on Data Mining,

authors:
Vitali Petsiuk
Abir Das
Kate Saenko

title: RISE: randomized input sampling for explanation of blackbox models

authors:
Ramprasaath R Selvaraju
Abhishek Das
Ramakrishna Vedantam
Michael Cogswell
Devi Parikh
Dhruv Batra

title: Grad-cam: Why did you say that

venue: arXiv preprint arXiv:1611.07450,

authors:
Ramprasaath R Selvaraju
Michael Cogswell
Abhishek Das
Ramakrishna Vedantam
Devi Parikh
Dhruv Batra

title: Grad-cam: Visual explanations from deep networks via gradient-based localization

venue: In Proceedings of the IEEE international conference on computer vision,

authors:
Dylan Slack
Sophie Hilgard
Emily Jia
Sameer Singh
Himabindu Lakkaraju

title: Fooling lime and shap: Adversarial attacks on post hoc explanation methods

venue: In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society,

authors:
Daniel Smilkov
Nikhil Thorat
Been Kim
Fernanda Viégas
Martin Wattenberg

title: Smoothgrad: removing noise by adding noise

venue: arXiv preprint arXiv:1706.03825,

authors:
Saurabh Srivastava
Sumit Gulwani
Jeffrey S Foster

title: Vs 3: Smt solvers for program verification

authors:
Pascal Sturmfels
Scott Lundberg
Su-In Lee

title: Visualizing the impact of feature attribution baselines. Distill, 2020

authors:
Mukund Sundararajan
Ankur Taly
Qiqi Yan

title: Axiomatic attribution for deep networks

venue: Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017,

authors:
Christian Szegedy
Wei Liu
Yangqing Jia
Pierre Sermanet
Scott Reed
Dragomir Anguelov
Dumitru Erhan
Vincent Vanhoucke
Andrew Rabinovich

title: Going deeper with convolutions

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Shawn Xu
Subhashini Venugopalan
Mukund Sundararajan

title: Attribution in scale and space

venue: arXiv preprint arXiv:2004.03383,

authors:
Xin Zhang
Armando Solar-Lezama
Rishabh Singh

title: Interpreting neural network judgments via minimal, stable, and symbolic corrections

heading: 1 INTRODUCTION

text:
Satisfiability Modulo Theory (SMT) solvers (Barrett & Tinelli, 2018) are routinely used for symbolic modeling and verifying correctness of software programs (Srivastava et al., 2009), and more recently they have also been used for verifying properties of deep neural networks (Katz et al., 2017). SMT Solvers in their current form are difficult to scale to large networks. Model explanation is one such domain where SMT solvers have been used but they are limited to very small sized networks (Gopinath et al., 2019; Ignatiev et al., 2019). The goal of our work is to address the issue of scalability of SMT solvers by using gradient information, thus enabling their use for different applications. In this work, we present a new application of SMT solvers for explaining neural network decisions.
Model explanation can be viewed as identifying a minimal set of features in a given input that is critical to a model’s prediction (Carter et al., 2018; Macdonald et al., 2019). Such a problem formulation for identifying a minimal set lends itself to the use of SMT solvers for this task. We can encode a neural network using real arithmetic (Katz et al., 2017) and use an SMT solver to optimize over the constraints to identify a minimal set of inputs that can explain the prediction. However, there are two key challenges in this approach. First, we cannot generate reliable explanations based on final model prediction as the minimal input is typically out of distribution. Second, solving such a formulation is challenging for SMT solvers as the decision procedures for solving these constraints have exponential complexity, and is further exacerbated by the large number of parameters in typical neural network models. Thus, previous approaches for SMT-based analysis of neural networks have been quite limited, and have only been able to scale to networks with few thousands of parameters.
To solve these challenges, instead of doing minimization by encoding the entire network, our approach takes advantage of the gradient information, specifically Integrated Gradients (IG) (Sundararajan et al., 2017), in lieu of encoding the deeper layers, and encodes a much simpler set of linear constraints
pertaining to the layer closest to the input. We encode the mathematical equations of a neural network as SMT constraints using the theory of Linear Real Arithmetic (LRA), and use z3 solver (Bjørner et al., 2015) as it additionally supports optimization constraints such as minimization. The SMT solver then finds a minimal subset (of input features) by performing minimization on these equations. Thus, our approach, which we refer to as SMUG, is able to scale Symbolic Methods Using Gradient information while still providing a faithful explanation of the neural network’s decision.
SMUG is built on two properties. First, based on the target prediction, SMUG uses gradient information propagated from the deeper layers to identify neurons that are important in the first layer, and only encodes those. For this, we use IG (Sundararajan et al., 2017) instead of relying on gradients alone. Second, for the optimization, a set of input pixels are determined to be relevant for prediction if they are able to activate the neurons deemed important, and maintain a fraction of their activation as the original (full) input image. Empirically, we observe good performance on visual and text classification tasks.
We evaluate SMUG on three datasets: MNIST (LeCun et al., 2010), ImageNet (Deng et al., 2009), and Beer Reviews (McAuley et al., 2012). We show that we can fully encode the minimal feature identification problem for a small feedforward network (without gradient-based neuron selection) for MNIST, but this full SMT encoding scales poorly for even intermediate sized networks. On ImageNet, we observe that our method performs better than Integrated Gradients (Sundararajan et al., 2017) and several strong baselines. Additionally, we observe that our approach finds significantly sparser masks (on average 17% of the original image size). Finally, we also show that our technique is applicable to text models where it performs competitively with other methods including SIS (Carter et al., 2018) and Integrated Gradients (Sundararajan et al., 2017).
This paper makes the following key contributions:
• We present a technique (SMUG) to encode the minimal input feature discovery problem for neural model explanation using SMT solvers. Our approach, which does masking on linear equations also overcomes the issue of handling out-of-distribution samples.
• Our approach uses gradient information to scale SMT-based analysis of neural networks to larger models and input features. Further, it also overcomes the issue of choosing a “baseline” parameter for Integrated Gradients (Kapishnikov et al., 2019; Sturmfels et al., 2020).
• We empirically evaluate SMUG on image and text datasets, and show that the minimal features identified by it are both quantitatively and qualitatively better than several baselines.
• To improve our understanding on saliency map evaluation, we show how the popular and widely used LSC metric (Dabkowski & Gal, 2017) can be gamed heuristically to generate explanations that are not necessarily faithful to the model (Sec. 5.2)

heading: 2 RELATED WORK

text:
SMT based symbolic techniques have been used for verifying neural network properties (Huang et al., 2017; Katz et al., 2017). Reluplex (Katz et al., 2017) extends the simplex method to handle ReLU functions by leveraging its piecewise linear property and presents an iterative procedure for gradual satisfaction of the constraints. (Huang et al., 2017) proposes a layer-wise analysis using a refinement-based approach with SMT solvers for verifying the absence of adversarial input perturbations. (Zhang et al., 2018) present a linear programming (LP) formulation again using the piecewise linear property of ReLU to find minimal changes to an input to change a network’s classification decisions. (Gopinath et al., 2019) uses Reluplex to learn input properties in the form of convex predicates over neuron activations, which in turn capture different behaviors of a neural network. While SMT based techniques from (Gopinath et al., 2019), (Ignatiev et al., 2019) have shown promising results, they only scale to Neural Network with 5000 nodes. Thus, scaling these approaches for larger neural networks and performing richer analysis based on global input features still remains a challenge. We present and demonstrate an approach that works for larger and more complex image and text models.
While most of the above SMT based techniques focus on verifying properties of deep networks, our work focuses on applying symbolic techniques to the related task of model explanation, i.e. to say where a model is “looking”, by solving for the input features responsible for a model’s prediction.
Some explanation techniques are model agnostic (i.e., black-box) while others are back-propagation based. Model agnostic (black-box) explanation techniques such as SIS, LIME (Alvarez-Melis & Jaakkola, 2018; Carter et al., 2018) have a similar formulation of the problem as ours in the sense that they perturb the input pixels by masking them and optimize to identify minimal regions affecting the performance of the model. This formulation can lead to evaluating the model on out of distribution samples (Hooker et al., 2019) with potential for adversarial attacks (Slack et al., 2020). In contrast, back-propagation based methods (Bach et al., 2015; Sundararajan et al., 2017; Selvaraju et al., 2016) examine the gradients of the model with respect to an input instance to determine pixel attribution. Our work builds on the IG method (Sundararajan et al., 2017). IG integrates gradients along the “intensity” path where the input (image or text embedding) is scaled from an information-less baseline (all zeros input, e.g., all black or random noise image) to a specific instance. This helps the model determine attribution at the pixel level. In our work, we use IG to determine important nodes in the first layer (closest to the input). The key improvement of our technique over IG is that, by using IG only on the first layer and then using SMT solver based solution to determine saliency on the input, we not only preserve faithfulness, but also overcome the issue of choosing an appropriate baseline image for IG (Kapishnikov et al., 2019; Sturmfels et al., 2020; Xu et al., 2020).

heading: 3 METHOD: SCALING SYMBOLIC METHODS USING GRADIENTS (SMUG)

text: We describe our approach SMUG, which combines attribution based on gradient information with an SMT-based encoding of the minimal input feature identification problem. We then show how to generate saliency maps from the predicted boolean mask for image and text applications.

heading: 3.1 CHALLENGES WITH ENCODING NEURAL NETWORKS FOR EXPLANATION

text:
Previous works (Katz et al., 2017) have shown it’s possible to use SMT solvers to encode the semantics of neural networks with specific activation functions, while others (Carter et al., 2018; Macdonald et al., 2019) have used the idea of minimality to identify importance of inputs in the context of model explanation. In particular, given a neural network Nθ (θ denotes the parameters), let X ∈ Rm×n denote an input image with m× n pixels, M ∈ {0, 1}m×n an unknown binary mask, and Li(·) the output (i.e., activations) of the ith layer. Also, let L|L|−1(Nθ(X)) denote the logits of the final layer. Previous works (Carter et al., 2018; Macdonald et al., 2019) have used the following formulation for explanation.
min( ∑ ij Mij) : argmaxL|L|−1(Nθ(X)) = argmaxL|L|−1(Nθ(M X))
However, there are two key problems with this formulation. First, the masked inputs found by solving this formulation are typically out of training distribution, so explanation techniques based on model confidence over a forward pass of the masked inputs (such as the above formulation) no longer produce reliable explanations. Second, using such a formulation directly with SMT solvers is not tractable as the number of constraints in the encoding grows linearly with increasing network size and SMT decision procedure for Non-linear Real Arithmetic is doubly exponential. Thus, when we apply such an encoding even for a small feed-forward network on MNIST dataset, the SMT solver does not scale well (Section 4, Appendix A).

heading: 3.2 OUR SOLUTION: COMBINING GRADIENTS WITH SYMBOLIC ENCODING

text:
Our proposal is to use gradient based attributions to overcome both these issues: 1) as a proxy for the deeper layers (for scalability), and 2) to identify a subset of neurons in the first layer capturing information crucial for the model’s prediction, and encode them using SMT which avoids performing a forward pass on out-of-distribution inputs.
Specifically, we use Integrated Gradients (IG) (Sundararajan et al., 2017) to score the neurons in order of relevance by treating the first layer activations (L1) as an input to the subsequent network. This assigns an attribution score to each hidden node where a node with a positive score means that the information captured by that neuron is relevant to the prediction and a node with a negative / zero score is considered irrelevant. More specifically, suppose F : Ra×b×c → [0, 1]d represents a deep network and x represents the input. F often refers to the full network (from the input to the
final softmax) but in our case, it refers to the nodes in the network second through the final layer. a, b, and c refer to the dimensions of the tensor input to the network (analogous to width, height, and channels) and d is the number of output nodes in the final softmax layer (number of classes). Integrated gradients are obtained by accumulating the gradients at all points along the straight line path from an “information-less” baseline x′ to the input x. The information-less baseline in this case is an all zeros tensor. The path can be parameterized as g(α) = x′ + α · (x− x′). IG is then given by Eq. 1.
IG(x) = (x− x′) ∫ 1 α=0 ∂F (g(α)) ∂g(α) dα (1)
Top-k Neurons. The second key idea of our approach is that we only consider activations with the highest positive attributions. Empirically, we still observe scaling issues with SMT when considering all first layer neurons (L1) with +ve attributions, so a method for picking a subset of neurons (top-k attributions) is important for practical application.
With these two ideas, we can now formulate our problem as follows. Let, Dk represent the set of k important nodes with the highest attributions in IG(Nθ(X)), and γ be a parameter which regulates how “active” the neurons remain after masking1.The goal now is to learn a mask M such that:
min( ∑ ij Mij) : L1(Nθ(M X))t > γ · L1(Nθ(X))t ∀t ∈ Dk (2)
Inducing Sparsity. IG ensures that the information captured by the neurons in Dk are relevant for the model’s prediction. Thus, the set of features corresponding to these neurons contributes towards the relevant information. Let’s denote this set as S. To filter less relevant features from S we introduce a notion of minimization. By inducing sparsity we wish to remove any false positives from S. Thus, we define "relevant features" as a minimal subset of features in S that causes the nodes in Dk to remain active. The notion being - if for a masked input the "important nodes" (w.r.t the original input) are active then the information relevant for the prediction of the original image is contained in the mask. Thus, the mask highlights features relevant for the model’s prediction.
Thus, this formalism allows us to come up with constraints which don’t involve a full forward pass of the masked image while at the same time keeping things scalable for the SMT solver.

heading: 3.3 SMT FORMULATION OF MINIMAL INPUT MASK DISCOVERY PROBLEM

text:
Given Dk, a set of k neurons with highest positive attributions in IG(Nθ(X)), our goal is to find a minimal mask such that the activations of these neurons are above some threshold (γ) times their original activation values. Eq. 3 shows the constraints for a minimal mask. The first set of constraint specifies that the unknown mask variable M can only have 0 and 1 as possible entries in the matrix. The second and third set of constraints encode the activation values of the first layer of network with corresponding masked and original inputs respectively. The fourth set of constraint states that the activations of these k neurons should be at least γ times the original activation values, and the final constraint adds the optimization constraint to minimize the sum of all the mask bits. Note that here we show the formulation for a feedforward network and a input X with 2 channels, but it can be extended to convolutional networks and an input with 3rd channel as well in a straightforward manner where the mask variables across the same channel share the same mask variable.
∃M : ∧
1≤i≤m,1≤j≤n (Mij == 0) ∨ (Mij == 1) ∧ ∀i∈Dk
omi = (W1(X M) + b1)i∧ ∀i∈Dk oi = (W1X + b1)i ∧ ∀i∈Dk omi > γ · oi ∧ minimize(ΣijMij) (3)

heading: 3.4 CONSTRUCTING SALIENCY MAP FROM BINARY MASK

text:
The SMT solver generates a minimal binary input mask by solving the constraints shown in Eq. 3. We further use the IG attribution scores for the hidden nodes in the first layer to assign importance
1This formulation is valid for networks with ReLU activations and would need to be modified for a different choice of activation such as tanh.
scores to mask pixels associated with that hidden node. A mask variable Mij that is assigned a value of 1 by SMT is assigned a score sij computed as:
sij = ∑
1≤p≤k
α(op)1receptive(op)(xij) ∀i, j : Mij = 1 (4)
where α(op) denotes the attribution score assigned by IG for neuron op and the indicator function denotes that pixel xij is present in the receptive field of op, i.e. it is present in the linear SMT equation used to compute on. These scores are then used to compute a continuous saliency map for an input (Appendix Fig. 9). Finally, to amplify the pixel differences for visualization purposes in gray scale, we scale the non-zero score values between 0.5 and 1.

heading: 3.5 HANDLING OUT-OF-DISTRIBUTION MASKED INPUTS

text: Several methods - (Alvarez-Melis & Jaakkola, 2018), (Petsiuk et al., 2018), (Fong & Vedaldi, 2017) optimize their masks by doing a forward pass on perturbed images that are potentially out-ofdistribution. However, SMUG performs forward pass of the masked image only till the first layer (instead of the final layer) to generate linear equations as described in Sec. 3.3. Since, the constraints are linear 6, minimization can be thought of as analogous to examining coefficients for a linear regression model (in linear regression important attributes are the ones with the largest coefficients). So, in our case we do indeed mitigate the out-of-distribution issue as the full forward pass of the masked input isn’t present.

heading: 4 EXPERIMENTAL SETUP

text:
Datasets. We empirically evaluate SMUG on two image datasets, MNIST (LeCun et al., 2010), and ImageNet (Deng et al., 2009), as well as a text dataset of Beer Reviews from (McAuley et al., 2012).
MNIST: We use the MNIST dataset to show the scalability of the full network encoding in SMT (presented in Section 3.1). We use a feedforward model consisting of one hidden layer with 32 nodes (ReLU activation) and 10 output nodes with sigmoid, one each for 10 digits (0 - 9). For 100 images chosen randomly from the validation set, the SMT solver could solve the constraint shown in Eq. 6 (returns SAT) for only 41 of the images. For the remaining 59 images, the solver returns UNKNOWN, which means the given set of constraints was too difficult for the solver to solve.
ImageNet: We use 3304 images (224× 224) with ground truth bounding boxes from the validation set of ImageNet. The images for which the model classification was correct and a ground truth bounding box annotation for the object class was available were chosen. We use the Inception-v1 model from (Szegedy et al., 2015) which classifies images into one of the 1000 ImageNet classes.
Beer Reviews: To evaluate SMUG on a textual task we consider the review rating prediction task on the Beer Reviews dataset2 consisting of 70k training examples, 3k validation and 7k test examples. Additionally, the dataset comes with ground truth annotations where humans provide the rationale (select words) that correspond to the rating and review. We train a 1D CNN model to predict the rating for the aroma of the beer on a scale from 0 to 1. This model is identical to the model used in (Carter et al., 2018) and consists of a convolution layer with 128 kernels followed by a ReLU, a fully connected layer, and a sigmoid. It achieves a validation mean square error of 0.032.
Metrics. Assessing the quality of the saliency maps, especially binary masks, is challenging. The change in confidence of the classifier (between the original and masked) image alone may not be a reliable measure since the masked input could fall out of the training distribution (Hooker et al., 2019). Instead, we use the metric proposed in (Dabkowski & Gal, 2017) shown in Eq. 5. This metric, which we term Log Sparsity Confidence difference (LSC) score, first finds the tightest bounding box that captures the entire mask, then computes confidence on the cropped box resized to the original image size (we use bilinear interpolation). This not only helps keep images closer to the training distribution, but also helps evaluate explanations without the need for groundtruth annotations. The LSC score is computed as:
LSC(a, c) = log(ã)− log(c), ã = max(0.05, a) (5) 2http://people.csail.mit.edu/taolei/beer/
where a is the fractional area of the rectangular cropped image and c is the confidence of the classifier for the true label on the cropped image. A saliency map that is compact and allows the model to still recognize the object would result in a lower LSC score. LSC captures model confidence as well as compactness of the identified salient regions, both of which are desirable when evaluating an explanation. The compactness in particular also makes it suitable for evaluating SMT based methods for the effect of minimization. We adapt LSC to also assess continuous valued saliency maps by, 1) setting a threshold on the continuous valued saliency map to convert them to a binary mask and 2) iterating over multiple thresholds (in steps) to identify the one that results in the best LSC score. We also report the fraction of images for which the mask generated by a given method is better (i.e. produces an equal or lower LSC score) than other methods, which we refer to as Win%.
Comparison methods. The final saliency mask for SMUG comes from Eq. 4 (Sec. 3.4). We compare this to the saliency maps, and bounding boxes from several baselines described below. SMUGbase is a variant of SMUG that does not perform SMT-based minimization. Here, in Eq. 2, we simply set Mij = 1 for each pixel xij that is in the receptive fields of the top-k neurons in the first layer (L1) selected by IG. We note here that in case of both SMUG and SMUGbase in the formulations in Eq. 2 and 3, we set k = 3000, γ = 0 for ImageNet, and k = 100, γ = 0 for text experiments (this choice is explored more in the supplementary material). IG corresponds to Integrated Gradients (Sundararajan et al., 2017) with the black image as a baseline. GRADCAM (Selvaraju et al., 2017) uses a weighted average of the CNN filters for saliency, with weights informed by the gradients. SIS refers to Sufficient Input Subset (Carter et al., 2018), which finds multiple disjoint subsets of input features (in decreasing order of relevance) which individually allow a confident classification. SIS did not scale for ImageNet and we only compare against it on the text dataset. GROUNDTRUTH corresponds to the baseline that uses human annotated bounding box, which capture the object corresponding to the image label. MAXBOX denotes maximal mask spanning the entire image. CENTERBOX uses a bounding box placed at the center of the image covering half of the image area. OPTBOX refers to a bounding box that approximately optimizes for LSC. The saliency metric in Eq. 5 relies on finding a single bounding box for an image. To find a box that directly maximizes the metric, we first discretize the image into subgrids of 10× 10 pixels; and then perform a brute force search by selecting 2 points on the grid (to represent opposite corners of a rectangle) and identify a subgrid with the best score.

heading: 5 RESULTS

heading: 5.1 IMAGENET

text:
As mentioned previously, when computing masks using SMUG, for ImageNet we set k = 3000 in Eq. 2 and 3. Further, each masking variable Mij is used to represent a 4× 4 grid of pixels instead of a single pixel (to reduce running time). Table 1 presents quantitative results reporting the median LSC score and Win% values. Fig. 1 present qualitative examples
IG vs SMUG and SMUGbase. From Table 1, we observe that SMUG and SMUGbase achieve a significantly better score (−1.26 and −1.23 resp.) compared to IG (−0.34). As observed in some qualitative examples (Fig. 1, Appendix D), SMUG tends to assign high scores to a much more localized set of pixels whereas IG distributes high scores more widely (spatially). As LSC metric favors compactness, which is desirable for human interpretability, it results in better scores for SMUG and SMUGbase.
Choice of baseline for IG. Another reason why SMUGbase and SMUG outperform IG is that, they apply IG to the first layer of the network (as opposed to the input/image layer). IG attribution in the pixel space is known to be noisy (Smilkov et al., 2017), further attributions produced by IG depend on the choice of the baseline (Kapishnikov et al., 2019; Xu et al., 2020; Sturmfels et al., 2020). The reason for this can be observed from Eq. 1. In Eq. 1, x′ represents the baseline “information-less” image. Based on this, the input dimensions close to the baseline receive very low attributions even though they might be important. i.e., if i, j denote pixel locations, when xi,j−x′i,j ≈ 0, the attribution IGi,j(x) ≈ 0 irrespective of how important the pixels are. For instance, black pixels (RGB value of (0, 0, 0)) will receive an exact 0 attribution for a black baseline. In fact, for any baseline, IG will be insensitive to the dimensions close to the baseline value. When IG is applied to the first layer activations however, the nodes with near 0 activations will by default receive less attribution. Thus, we believe that 0 activations in the first layer is a more natural baseline for IG for ReLU networks, which is quantitatively observable in better LSC scores.
SMUG vs SMUGbase.
Based on the LSC scores in Table 1, SMUG narrowly outperforms SMUGbase. Recall however, that SMUG is a sparser version of SMUGbase obtained from the minimization constraints of the SMT solver (Eqns. 2 and 3). LSC scores use the sparsity of the bounding boxes as a proxy for the sparsity of the saliency map. As we can see in Fig. 1(a) and 1(b) that the bounding boxes for SMUG and SMUGbase have similar sizes even though the former produces much sparser saliency maps. This causes both SMUG and SMUGbase to receive similar LSC scores. Thus, the LSC scores alone don’t reveal the full picture and we need to consider the sparsity as well. This is defined as the fraction of the pixels with non-zero attributions to the total number of pixels in the image. We find that the average SMUGbase mask has a sparsity of 43% while the average SMUG mask has a sparsity of just 17%. This is also evident from the examples in Fig. 1. SMUG and SMUGbase receive similar LSC scores which suggests that the symbolic encoding successfully retains pixels relevant to the prediction while removing about 66% pixels from SMUGbase.
GROUNDTRUTH, CENTERBOX. Based on qualitative examples Figs. 1, we can observe that in almost all cases the object is at the center of the image. Hence, CENTERBOX is likely to capture some part of the image. Further, a fair number of objects are large covering much of the image e.g., Fig. 1 Macaque, Collie, Robin, Dalmation. In these cases, the groundtruth bounding boxes are also large to fully cover all pixels corresponding to the object. In contrast, SMUG saliency maps are more compact for both large and small objects, and hence achieve a better LSC score.

heading: 5.2 DISCUSSION: ANALYZING THE LSC METRIC AND OPTBOX

text:
The LSC metric makes a trade-off when optimizing for both compactness and confidence. To analyze this we look at several qualitative examples (Fig. 2) of the bounding boxes identified by the OPTBOX brute-force approach to optimize the LSC metric. We observe that OPTBOX often finds bounding boxes that are very small, typically a sufficiently discriminative region or pattern in the image (e.g., typewriter keyboard, paddlewheel, vine snake in Fig. 2), or the full object if the object is itself small (e.g., basketball, plunger). In all these cases we find that SMUG highlights several other aspects of the object as well (typewriter’s tape; the dog’s eyes, nose and ears, etc.)
OPTBOX can game the LSC metric (Dabkowski & Gal, 2017) to favor compact regions containing redundant features. We design an experiment where we introduce redundant features and compare the explanations generated by various methods. Fig. 3(c) contains 2 cats - catbig and catsmall where catsmall is catbig scaled to quarter its size. This image is classified correctly with a confidence of 0.84. If catsmall is removed from Fig. 3(c) we get Fig. 3(a) which is classified with a confidence of 0.83 (a marginal drop of 0.01 in the confidence score) and if we remove catbig we get Fig. 3(b) which is classified with a confidence of 0.15 (the confidence drops by 0.68). Thus, catsmall is redundant in Fig. 3(c) as its contribution to the prediction is nominal. Hence, the expectation is that a good saliency method would assign greater attribution to catbig than catsmall. In this respect, both SMUG and IG capture the model’s behavior correctly. Additionally, SMUG’s attributions lie on the catbig’s face while IG’s lie on the edges (possibly due to a sharp change in contrast). OPTBOX, however, attributes the model’s prediction to catsmall. This behavior can be explained using Eq. 5. Essentially, OPTBOX employs a brute force search to look for a region as small as possible and classified with a good enough confidence by the model so as to maximize Eq. 5. And by doing so it highlights regions with tightly packed redundant features (catsmall in this case). As argued in (Dabkowski & Gal, 2017) LSC does capture certain desirable attributes of saliency maps which makes it a popular choice to evaluate these explanation methods, but in this work we show that saliency methods can be designed to fool this metric.

heading: 5.3 TEXT DATASET: BEER REVIEWS

text: We present randomly selected qualitative examples from the test set comparing with other methods including SIS, IG, and GROUNDTRUTH in Fig. 4 and Appendix C. The solution of SIS consists of multiple disjoint set of words of varying relevance. A saliency map is constructed by scoring the words in the sets between (0,1] on the basis of relevance of the set. However, evaluation is harder. Unlike images where the masked image can be cropped and resized as input to compute the LSC metric, the same strategy cannot be followed on the text model. Specifically, ImageNet models are trained with extensive data-augmentation including random crops and resizing, and the modified image is less likely to be out-of-distribution. Whereas in the case of text, this model doesn’t employ any form of meaningful augmentation, and the masked text is much more likely to come from a distribution that has not been seen during training. Hence, LSC is not applicable here.

heading: 6 CONCLUSION

text: We present an approach that uses SMT solvers for computing minimal input features that are relevant for a neural network prediction. In particular, it uses attribution scores from Integrated Gradients to find a subset of important neurons in the first layer of the network, which allows the SMT encoding of constraints to scale to larger networks for finding minimal input masks. We evaluate our technique to analyze models trained on image and text datasets and show that the saliency maps generated by our approach are competitive or better than existing approaches and produce sparser masks.

heading: A MNIST

text:
In this section, we present some more details about our experiments with MNIST using the full SMT encoding from Eq. 1 in Sec 3.1.
SMT solvers can be used to encode the semantics of a neural network (Katz et al., 2017). In particular, given a fully connected neural network with n hidden layers, weightsW = {W1,W2 . . . ,Wn}, biases B = {b1, b2 . . . , bn}, activation function φ, and final layer softmax σ, we can use the SMT theory of nonlinear real arithmetic to obtain a symbolic encoding of the network. Let X ∈ Rm×n denote an input image with m × n pixels, M ∈ {0, 1}m×n an unknown binary mask, Li the output (i.e., activations) of the ith layer (L0 = X is the input) and α(pj) the output of jth logit in the final layer:
Li ≡ φ(WiLi−1 + bi) α(pj ,W,B,X) ≡ σj(WnLn−1 + bn)
Given this symbolic encoding, we can encode the minimal input mask generation problem as:
∃M : minimize(ΣijMij)∧α(plabel,W,B,M X) > α(pl,W,B,M X) ∀l 6= label (6)
where plabel and pl refer to the logits corresponding to the true label and the other labels respectively. The number of constraints grow with increasing network size and SMT decision procedure for Nonlinear Real Arithmetic is doubly exponential. Even for piecewise ReLU networks, SMT decision procedures for Linear Real Arithmetic combine simplex-based methods (exponential complexity) with other decision procedures such as Boolean logic (NP-complete complexity), which causes the solving times to grow dramatically with network size. When we apply this encoding even for a small feed-forward network on MNIST dataset, the SMT solver does not scale well (Section 4). This motivates our proposed approach for using gradient information to simplify the SMT constraints.
Table 2 shows the SMT solver runtimes and whether the constraints were solved (SAT). We observe that with a timeout of 60 minutes, the SMT solver could solve the full constraints for only 34 of the 100 images. Another interesting point to observe is that the solver could not solve any of the instances for digits 0 and 3. We also show some of the minimal masks generated by the SMT solver for few MNIST images in Figure 5.

heading: B HYPERPARAMETER CHOICES

text:
In our proposed approach, the choice of top-k, and γ (Eqns. 3, 4, 5) have an effect on the final quality of the explanations, and the time it takes for the solver to identify the mask. This section presents quantitative and qualitative comparisons for different choices of top-k and γ.
B.1 QUANTITATIVE COMPARISON FOR DIFFERENT CHOICES OF TOP-k AND γ
Fig. 6 presents quantitative comparisons for different choices of top-k and γ.
top-k We analyze images with top-k ∈ {500, 1000, 3000, 5000}. Increasing the k value increases the receptive field and the discovered input masks also grow in size with increasing values of k. Figure 6 shows how the solver runtime and the mask size vary with k. As expected, larger k values results in larger number of constraints and therefore larger solving times as well as larger mask sizes.
Gamma We analyze the effect of γ ∈ {0.0, 0.5, 0.9}, also shown in in Fig. 6. We observe that by decreasing gamma values, the masks become sparser. The key reason behind this is that with smaller gamma values, the SMTsolver is less constrained to maintain the original neural activations for the selected neurons, and hence can ignore additional input pixels that do not have a large effect. It is also noteworthy to notice that the solver run-time increases with decreasing value of γ.
B.2 TOP-K VS γ ON IMAGENET - QUALITATIVE EXAMPLES
Fig. 7 presents qualitative examples of the saliency maps on Imagenet examples for different choices of top-k and γ.

heading: C QUALITATIVE TEXT EXAMPLES

text: Fig. 8 presents additional examples comparing the output of SMUG with SIS, SMUGbase, and IG on the Beer Reviews dataset.

heading: D ADDITIONAL QUALITATIVE IMAGE EXAMPLES: SMUG

text:
Fig. 9 presents boolean masks and the saliency maps produced by SMUG on several ImageNet examples. Fig. 10 and 11 present additional examples comparing the saliency masks and bounding box (for LSC) produced by SMUG, SMUGbase, and IG.
D.1 BIASES
Model explanation techniques can also be particularly useful in studying model biases. Fig. 12 shows some examples where the model correctly predicts the class as “parallel bars” but it appears to actually focus more on the person leaping over the bar to make the prediction as opposed to looking at the bar itself. These can help us understand and identify cases where the model has developed a bias (in this case, based on training data).

heading: E SMT CONSTRAINTS

text:
In this section, we present an example set of SMT constraints obtained by our technique for an example image from ImageNet. For brevity, we show the set of constraints for k = 5 and γ = 0. As mentioned in Sec 5.1, each masking variable Mij corresponds to a 4× 4 grid of pixels, where the grid is denoted by Xi:i+3,j:j+3. For example, the mask variable M132,135 corresponds to the pixel grid X132:135,132:135. Following Eq. 3, the SMT constraints corresponding for top k = 5 IG positive attributions in the first layer are given by:
99.53X132,132 − 58.37X132,136 + 4.88X132,140 − 141.25X136,132 + 639.97X136,136 + 10.29X136,140 − 9.66X140,132 + 20.30X140,136 − 25.19X140,140 − 0.58 > 0
−270.67M120,150 + 101.23M142,144 + 10.38M113,124 + 207.98M122,121 + 640.64M121,121 − 100.72M121,126 + 25.06M121,165 − 75.49M121,156 + 75.47M112,154 − 0.36 > 0
2925.38X144,132 − 395.09X144,136 + 81.61X148,132 − 999.88X148,136 − 82.70X152,132 + 17.08X152,136 + 0.21 > 0
−20.87X76,80+8.40X76,84−122.72X80,80+929.71X80,84+85.52X84,80+138.99X84,84−0.01 > 0
231.34X168,148+722.71X168,152+80.18X172,148+663.96X172,152+5.37X176,148+4.63X176,152+ 0.12 > 0


summaries: This paper tackles the task of pose prediction and takes a render-and-compare approach. However, instead of rendering pixel colors, the key insight is to render features -- each mesh vertex is associated with 3D (learned) features which are encouraged to match computed 2D image features. This 'neural mesh' representation allows pose inference via SGD as one can optimize for pose s.t. the rendered features best match the image features, and is also robust to foreground occlusion. The paper demonstrates results on ObjectNet3d and PASCAL3D+ where the proposed approach is shown to be more robust to occlusion and also better at precise pose estimation.

summaries: The paper presents a novel approach for 3d pose estimation by combining render-and-compare (analysis-by-synthesis) and contrastive feature learning. The key idea is to render and compare learned latent features instead of synthesized RGB colors to optimize 6D pose parameters. The proposed method learns latent feature vectors on a template mesh as well as target images via backbone neural networks such that matched regions have similar features while latent features are as distinctive as possible. The paper evaluates the novel formulation on  PASCAL3D+, the occluded PASCAL3D+, and ObjectNet3D dataset, demonstrating the render-and compare optimization with the proposed approach is more robust to appearance change and partial occlusions.

summaries: The authors propose a novel 3D neural mesh model of objects that is generative. They demonstrate that standard deep learning approaches to 3D pose estimation are highly sensitive to partial occlusion. Since their method works in a render and compare manner, it enables the method to be more robust to artifacts in general and partial occlusion in particular. They also achieve a highly competitive 3D pose estimation performance on popular dataset. They go on to show that even very crude prototypical approximation of the object geometry using a cuboid.

abstractText: 3D pose estimation is a challenging but important task in computer vision. In this work, we show that standard deep learning approaches to 3D pose estimation are not robust when objects are partially occluded or viewed from a previously unseen pose. Inspired by the robustness of generative vision models to partial occlusion, we propose to integrate deep neural networks with 3D generative representations of objects into a unified neural architecture that we term NeMo. In particular, NeMo learns a generative model of neural feature activations at each vertex on a dense 3D mesh. Using differentiable rendering we estimate the 3D object pose by minimizing the reconstruction error between NeMo and the feature representation of the target image. To avoid local optima in the reconstruction loss, we train the feature extractor to maximize the distance between the individual feature representations on the mesh using contrastive learning. Our extensive experiments on PASCAL3D+, occluded-PASCAL3D+ and ObjectNet3D show that NeMo is much more robust to partial occlusion and unseen pose compared to standard deep networks, while retaining competitive performance on regular data. Interestingly, our experiments also show that NeMo performs reasonably well even when the mesh representation only crudely approximates the true object geometry with a cuboid, hence revealing that the detailed 3D geometry is not needed for accurate 3D pose estimation. The code is publicly available at https://github.com/Angtian/NeMo.

authors:
Yutong Bai
Angtian Wang
Adam Kortylewski
Alan Yuille

title: Coke: Localized contrastive learning for robust keypoint detection

venue: arXiv preprint arXiv:2009.14115,

authors:
Volker Blanz
Thomas Vetter

title: A morphable model for the synthesis of 3d faces

venue: In Proceedings of the 26th annual conference on Computer graphics and interactive techniques,

title: Face recognition based on fitting a 3d morphable model

venue: IEEE Transactions on pattern analysis and machine intelligence,

authors:
Xu Chen
Zijian Dong
Jie Song
Andreas Geiger
Otmar Hilliges

title: Category level object pose estimation via neural analysis-by-synthesis

venue: arXiv preprint arXiv:2008.08145,

authors:
Bernhard Egger
Sandro Schönborn
Andreas Schneider
Adam Kortylewski
Andreas MorelForster
Clemens Blumer
Thomas Vetter

title: Occlusion-aware 3d morphable models and an illumination prior for face image analysis

venue: International Journal of Computer Vision,

authors:
Ross Girshick
Pedro Felzenszwalb
David McAllester

title: Object detection with grammar models

venue: Advances in Neural Information Processing Systems,

authors:
Raia Hadsell
Sumit Chopra
Yann LeCun

title: Dimensionality reduction by learning an invariant mapping

venue: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06),

authors:
Tengda Han
Weidi Xie
Andrew Zisserman

title: Video representation learning by dense predictive coding

venue: In Proceedings of the IEEE International Conference on Computer Vision Workshops,

authors:
Kaiming He
Haoqi Fan
Yuxin Wu
Saining Xie
Ross Girshick

title: Momentum contrast for unsupervised visual representation learning

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

title: Model-based image analysis for forensic shoe print recognition

venue: PhD thesis, University of Basel,

authors:
Adam Kortylewski
Ju He
Qing Liu
Alan L Yuille

title: Compositional convolutional neural networks: A deep architecture with innate robustness to partial occlusion

authors:
Adam Kortylewski
Qing Liu
Angtian Wang
Yihong Sun
Alan Yuille

title: Compositional convolutional neural networks: A robust and interpretable model for object recognition under occlusion

venue: International Journal of Computer Vision, pp

authors:
Adam Kortylewski
Qing Liu
Huiyu Wang
Zhishuai Zhang
Alan Yuille

title: Combining compositional models and deep networks for robust object classification under occlusion

venue: In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1333–1341,

authors:
Vincent Lepetit
Francesc Moreno-Noguer
Pascal Fua

title: Epnp: An accurate o (n) solution to the pnp problem

venue: International journal of computer vision,

authors:
Yi Li
Gu Wang
Xiangyang Ji
Yu Xiang
Dieter Fox

title: Deepim: Deep iterative matching for 6d pose estimation

venue: In Proceedings of the European Conference on Computer Vision (ECCV),

authors:
Tsung-Yi Lin
Michael Maire
Serge Belongie
James Hays
Pietro Perona
Deva Ramanan
Piotr Dollár
C Lawrence Zitnick

title: Microsoft coco: Common objects in context

venue: In European conference on computer vision,

authors:
C-P Lu
Gregory D Hager
Eric Mjolsness

title: Fast and globally convergent pose estimation from video images

venue: IEEE transactions on pattern analysis and machine intelligence,

authors:
Arsalan Mousavian
Dragomir Anguelov
John Flynn
Jana Kosecka

title: 3d bounding box estimation using deep learning and geometry

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Aaron van den Oord
Yazhe Li
Oriol Vinyals

title: Representation learning with contrastive predictive coding

venue: arXiv preprint arXiv:1807.03748,

authors:
Georgios Pavlakos
Xiaowei Zhou
Aaron Chan
Konstantinos G Derpanis
Kostas Daniilidis

title: 6-dof object pose from semantic keypoints

venue: IEEE international conference on robotics and automation (ICRA),

authors:
Sida Peng
Yuan Liu
Qixing Huang
Xiaowei Zhou
Hujun Bao

title: Pvnet: Pixel-wise voting network for 6dof pose estimation

authors:
Nikhila Ravi
Jeremy Reizenstein
David Novotny
Taylor Gordon
Wan-Yen Lo
Justin Johnson
Georgia Gkioxari

title: Accelerating 3d deep learning with pytorch3d

authors:
Sandro Schönborn
Bernhard Egger
Andreas Morel-Forster
Thomas Vetter

title: Markov chain monte carlo for automated face image analysis

authors:
Florian Schroff
Dmitry Kalenichenko
James Philbin

title: Facenet: A unified embedding for face recognition and clustering

title: Improved deep metric learning with multi-class n-pair loss objective

venue: In Advances in neural information processing systems,

authors:
Chen Song
Jiaru Song
Qixing Huang

title: Hybridpose: 6d object pose estimation under hybrid representations

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Hao Su
Charles R Qi
Yangyan Li
Leonidas J Guibas

title: Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views

venue: In Proceedings of the IEEE International Conference on Computer Vision, pp

authors:
Martin Sundermeyer
Zoltan-Csaba Marton
Maximilian Durner
Manuel Brucker
Rudolph Triebel

title: Implicit 3d orientation learning for 6d object detection from rgb images

authors:
Ryan Szeto
Jason J Corso

title: Click here: Human-localized keypoints as guidance for viewpoint estimation

authors:
Shubham Tulsiani
Jitendra Malik

title: Viewpoints and keypoints

authors:
Angtian Wang
Yihong Sun
Adam Kortylewski
Alan L. Yuille

title: Robust object detection under occlusion with context-aware compositionalnets

authors:
He Wang
Srinath Sridhar
Jingwei Huang
Julien Valentin
Shuran Song
Leonidas J Guibas

title: Normalized object coordinate space for category-level 6d object pose and size estimation

authors:
Zhirong Wu
Yuanjun Xiong
Stella X Yu
Dahua Lin

title: Unsupervised feature learning via nonparametric instance discrimination

authors:
Yu Xiang
Roozbeh Mottaghi
Silvio Savarese

title: Beyond pascal: A benchmark for 3d object detection in the wild

venue: In IEEE Winter Conference on Applications of Computer Vision (WACV),

authors:
Yu Xiang
Wonhui Kim
Wei Chen
Jingwei Ji
Christopher Choy
Hao Su
Roozbeh Mottaghi
Leonidas Guibas
Silvio Savarese

title: Objectnet3d: A large scale database for 3d object recognition

venue: In European Conference Computer Vision (ECCV),

authors:
Sergey Zakharov
Ivan Shugurov
Slobodan Ilic

title: Dpod: 6d pose object detector and refiner

authors:
M Zeeshan Zia
Michael Stark
Konrad Schindler

title: Explicit occlusion modeling for 3d object class representations

authors:
Xingyi Zhou
Arjun Karpur
Linjie Luo
Qixing Huang

title: Starmap for category-agnostic keypoint and viewpoint estimation

heading: 1 INTRODUCTION

text:
Object pose estimation is a fundamentally important task in computer vision with a multitude of realworld applications, e.g. in self-driving cars, or partially autonomous surgical systems. Advances in the architecture design of deep convolutional neural networks (DCNNs) Tulsiani & Malik (2015); Su et al. (2015); Mousavian et al. (2017); Zhou et al. (2018) increased the performance of computer vision systems at 3D pose estimation enormously. However, our experiment shows current 3D pose estimation approaches are not robust to partial occlusion and when objects are viewed from a previously unseen pose. This lack of robustness can have serious consequences in real-world applications and therefore needs to be addressed by the research community.
In general, recent works follow either of two approaches for object pose estimation: Keypoint-based approaches detect a sparse set of keypoints and subsequently align a 3D object representation to the detection result. However, due to the sparsity of the keypoints, these approaches are highly vulnerable when the keypoint detection result is affected by adverse viewing conditions, such as partial occlusion. On the other hand, rendering-based approaches utilize a generative model, that is built on a dense 3D mesh representation of an object. They estimate the object pose by reconstructing the input image in a render-and-compare manner (Figure 1). While rendering-based approaches can be more robust to partial occlusion Egger et al. (2018), their core limitation is that they model objects in terms of image intensities. Therefore, they pay too much attention to object details that are not relevant for the 3D pose estimation task. This makes them difficult to optimize Blanz & Vetter (2003); Schönborn et al. (2017), and also requires a detailed mesh representation for every shape variant of an object class (e.g. they need several types of sedan meshes instead of using one prototypical type of sedan).
In this work, we introduce NeMo a rendering-based approach to 3D pose estimation that is highly robust to partial occlusion, while also being able to generalize to previously unseen views. Our key idea is to learn a generative model of an object category in terms of neural feature activations, instead of image intensities (Figure 1). In particular, NeMo is composed of a prototypical mesh representation of the object category and feature representations at each vertex of the mesh. The feature representations are learned to be invariant to instance specific details (such as shape and color variations) that are not relevant for the 3D pose estimation task. Specifically, we use contrastive learning He et al. (2020); Wu et al. (2018); Bai et al. (2020) to ensure that the extracted features of an object are distinct from each other (e.g. the features of the front tire of a car are different from those of the back tire), while also being distinct from non-object features in the background. Furthermore, we train a generative model of the feature activations at every vertex of the mesh representation. During inference, NeMo estimates the object pose by reconstructing a target feature map with using render-and-compare and gradient-based optimization w.r.t. the 3D object pose parameters.
We evaluate NeMo at 3D pose estimation on the PASCAL3D+ Xiang et al. (2014) and the ObjectNet3D Xiang et al. (2016) dataset. Both datasets contain a variety of rigid objects and their corresponding 3D CAD models. Our experimental results show that NeMo outperforms popular approaches such as Starmap Zhou et al. (2018) at 3D pose estimation by a wide margin under partial occlusion, and performs comparably when the objects are not occluded. Moreover, NeMo is exceptionally robust when objects are seen from a viewpoint that is not present in the training data. Interestingly, we also find that the mesh representation in NeMo can simply approximate the true object geometry with a cuboid, and still perform very well. Our main contributions are:
1. We propose a 3D neural mesh model of objects that is generative in terms of contrastive neural network features. This representation combines a prototypical geometric representation of the object category with a generative model of neural network features that are invariant to irrelevant object details.
2. We demonstrate that standard deep learning approaches to 3D pose estimation are highly sensitive to out-of-distribution data including partial occlusions and unseen poses. In contrast, NeMo performs 3D pose estimation with exceptional robustness.
3. In contrast to other rendering-based approaches that require instance-specific mesh representations of the target objects, we show that NeMo achieves a highly competitive 3D pose estimation performance even with a very crude prototypical approximation of the object geometry using a cuboid.

heading: 2 RELATED WORK

text:
Category-Level Object Pose Estimation. Category-Level object pose estimation has been well explored by the research community. A classical approach as proposes by Tulsiani & Malik (2015) and Mousavian et al. (2017) was to formulate object pose estimation as a classification problem. Another common category-level object pose estimation approach involves a two-step process Szeto & Corso (2017); Pavlakos et al. (2017): First, semantic keypoints are detected interdependently and subsequently a Perspective-n-Point problem is solved to find the optimal 3D pose of an object mesh
Lu et al. (2000); Lepetit et al. (2009). Zhou et al. (2018) further improved this approach by utlizing depth information. Recent work Wang et al. (2019); Chen et al. (2020) introduced render-andcompare to for category-level pose estimation. However, both approaches used pixel-level image synthesis and required detailed mesh models during training.In contrast, NeMo preforms renderand-compare on the level of contrastive features, which are invariant to intra-category nuisances, such as shape and color variations. This enables NeMo to achieve accurate 3D pose estimation results even with a crude prototypical category-level mesh representation.
Pose Estimation under Partial Occlusion. Keypoint-based pose estimation methods are sensitive to outliers, which can be caused by partial occlusion Pavlakos et al. (2017); Sundermeyer et al. (2018). Some rendering-based approaches achieve satisfactory results on instance-level pose estimation under partial occlusion Song et al. (2020); Peng et al. (2019); Zakharov et al. (2019); Li et al. (2018). However, these approaches render RGB images or use instance-level constraints, e.g. pixellevel voting, to estimate object pose. Therefore, these approaches are not suited for category-level pose estimation. To the best of our knowledge, NeMo is the first approach that performs categorylevel pose estimation robustly under partial occlusion.
Contrastive Feature Learning. Contrastive learning is widely used in deep learning research. Hadsell et al. (2006) proposed an intuitive tuple loss, which was later extended to triplets and NPair tuples Schroff et al. (2015); Sohn (2016). Recent contrastive learning approaches showed high potential in unsupervised learning Wu et al. (2018); He et al. (2020). Oord et al. (2018); Han et al. (2019) demonstrated the effectiveness of feature-level contrastive losses in representation learning. Bai et al. (2020) proposed a keypoint detection framework by optimizing the feature representations of keypoints with a contrastive loss. In this work, we use contrastive feature learning to encourage the feature extraction backbone to extract locally distinct features. This, in turn, enables 3D pose estimation by simply optimizing the neural reconstruction loss with gradient descent.
Robust Vision through Analysis-by-Synthesis. In a broader context, our work relates to a line of work in the computer vision literature, which demonstrate that the explicit modeling of the object structure significantly enhances the robustness of computer vision models, e.g. at 3D pose estimation Zeeshan Zia et al. (2013), face reconstruction Egger et al. (2018) and human detection Girshick et al. (2011) under occlusion. More specifically, our work builds on a recent line of work that introduced a neural analysis-by-synthesis approach to vision Kortylewski et al. (2020b) and demonstrated its effectiveness in occlusion-robust image classification Kortylewski et al. (2020a;c) and object detection Wang et al. (2020). Our work significantly extends neural analysis-by-synthesis to include an explicit 3D object representation, instead of 2D template-like object representations. This enables our model to achieve state-of-the-art robustness at pose estimation through neural render-and-compare.

heading: 3 NEMO: A 3D GENERATIVE MODEL OF NEURAL FEATURES

text: We denote a feature representation of an input image I as Φ(I) = F l ∈ RH×W×D. Where l is the output of a layer l of a DCNN Φ, with D being the number of channels in layer l. f li ∈ RD is a feaure vector in F l at position i on the 2D lattice P of the feature map. In the remainder of this section we omit the superscript l for notational simplicity because this is fixed a-priori in our model.

heading: 3.1 NEURAL RENDERING OF FEATURE MAPS

text:
Similar to other graphics-based generative models, such as e.g. 3D morphable models Blanz & Vetter (1999); Egger et al. (2018), our model builds on a 3D mesh representation that is composed of a set of 3D vertices Γ = {r ∈ R3|r = 1, . . . , R}. For now, we assume the object mesh to be given at training time but we will relax this assumption in later sections. Different from standard graphics-based generative models, we do not store RGB values at each mesh vertex r but instead store feature vectors Θ = {θr ∈ RD|r = 1, . . . , R}. Using standard rendering techniques, we can use this 3D neural mesh model N = {Γ,Θ} to render feature maps:
F̄ (m) = <(N,m) ∈ RH×W×D, (1) where m are the camera parameters for projecting the neural mesh representation (Figure 2).

heading: 3.2 NEURAL MESH MODELS

text:
Neural Mesh Models are probabilistic generative models of neural feature activations. Hence, our goal is to learn a generative model p(F |Ny) of the real-valued feature activations F of an object
class y by leveraging a 3D neural mesh representation Ny . Assuming that the 3D pose m of the object in the input image is known, we define the likelihood of the feature representation F as:
p(F |Ny,m,B) = ∏ i∈FG p(fi|Ny,m) ∏ i′∈BG p(fi′ |B). (2)
The foreground FG is the set of all positions on the 2D lattice P of the feature map F that are covered by the rendered neural mesh model. We compute FG by projecting the 3D vertices of the mesh model Γy into the image using the ground truth camera pose m to obtain the 2D locations of the visible vertices in the image FG = {st ∈ R2|t = 1, . . . , T}. We define foreground feature likelihoods to be Gaussian distributed:
p(fi|Ny,m) = 1
σr √ 2π exp
( − 1
2σ2r ‖fi − θr‖2
) . (3)
Note that the correspondence between the feature vector fi in the feature map F and the vector θr on the neural mesh model is given by the 2D projection of Ny with camera parameters m. Those features that are not covered by the neural mesh model BG = P \ {FG}, i.e. are located in the background, are modeled by a Gaussian likelihood:
p(fi′ |B) = 1
σ √ 2π exp
( − 1
2σ2 ‖fi′ − β‖2
) , (4)
with mixture parameters B = {β, σ}.

heading: 3.3 TRAINING USING MAXIMUM LIKELIHOOD AND CONTRASTIVE LEARNING

text:
During training we want to optimize two objectives: 1) The parameters of the generative model as defined in Equation 2 should be optimized to achieve maxmimum likelihood on the training data. 2) The backbone used for feature extraction ψ should be optimized to make the individual feature vectors as disctinct from each other as possible.
Maximum likelihood estimation of the generative model. We optimize the parameters of the generative model to minimize the negative log-likelihood of our model (Equation 2):
LML(F,Ny,m,B) =− ln p(F |Ny,m,B) (5) =− ∑ i∈FG ln ( 1 σr √ 2π ) − 1 2σ2r ‖fi − θr‖2 (6)
+ ∑ i′∈BG ln ( 1 σ √ 2π ) − 1 2σ2 ‖fi′ − β‖2 (7)
If we constrain the variances such that {σ2 = σ2r = 1|∀r} then the maximum likelihood loss reduces to:
LML(F,Ny,m,B) = −C ∑ i∈FG ‖fi − θr‖2 + ∑ i′∈BG ‖fi′ − β‖2, (8)
where C is a constant scalar.
Contrastive learning of the feature extractor. The general idea of the contrastive loss is to train the feature extractor such that the individual feature vectors on the object are distinct from each other, as well as from the background:
LFeature(F,FG) = − ∑ i∈FG ∑ i′∈FG\{i} ‖fi − fi′‖2 (9)
LBack(F,FG,BG) = − ∑ i∈FG ∑ j∈BG ‖fi − fj‖2. (10)
The contrastive feature loss LFeature encourages the features on the object to be distinct from each other (e.g. the feature vectors at the front tire of a car are different from those of the back tire). The contrastive background loss LBack encourages the features on the object to be distinct from the features in the background. The overall loss used to train NeMo is:
L(F,Ny,m,B) = LML(F,Ny,m,B) + LFeature(F,FG) + LBack(F,FG,BG) (11)

heading: 3.4 ROBUST 3D POSE ESTIMATION WITH RENDER AND COMPARE

text:
After training the feature extractor and the generative model in NeMo, we apply the model for estimating the camera pose parameters b. In particular we aim to optimize the model likelihood from Equation 2 w.r.t. to the camera parameters in a render-and-compare manner. Following related work on robust inference with generative models Kortylewski (2017); Egger et al. (2018) we optimize a robust model likelihood: p(F |Ny,m,B, zi) =
∏ i∈FG [p(fi|Ny,m)p(zi=1)]zi [p(fi|B)p(zi=0)](1−zi) ∏ i′∈BG p(fi′ |B). (12)
Here zi ∈ {0, 1} is a binary variable and p(zi=1) and p(zi=0) are the prior probabilities of the respective values. Here zi is a binary variable that allows the background model p(fi|B) to explain those locations in the feature map F that are in the foreground region FG, but which the foreground model (fi|Ny,m) cannot explain well. A primary purpose of this mechanism is to make the cost function robust to partial occlusion. Figure 2 illustrates the inference process. Given an initial camera pose estimate we use the Neural Mesh Model to render a feature map F̄ and evaluate the reconstruction loss in the foreground region FG (foreground score map), as well as the reconstruction error when using the background model only (background score map). Pixel-wised comparison of foreground score and background score yield the occlusion map Z = {zi ∈ {0, 1}|∀i ∈ P}. The map Z indicates where feature vectors are explained by either the foreground or background model.
A fundamental benefit of our Neural Mesh Models is that, they are generative on the level of neural feature activations. This makes the overall reconstruction loss very smooth compared to related works who are generative on the pixel level. Therefore, NeMo can be optimized w.r.t. the pose parameters with standard stochastic gradient descent. We visualize the loss as a function of the individual pose parameters in Figure 2. Note that the losses are generally very smooth and contain one clear global optimum. This is in stark contrast to the optimization of classic generative models at the level of RGB pixels, which often requires complex hand designed initialization and optimization procedures to avoid the many local optima of the reconstruction loss Blanz & Vetter (2003); Schönborn et al. (2017).

heading: 4 EXPERIMENT

text: We first describe the experimental setup in Section 4.1. Subsequently, we study the performance of NeMo at 3D pose estimation in Section 4.2 and study the effect of crudely approximating the object geometry within NeMo single 3d cuboid, that one cuboid represent all object in each category, and multiple 3d cuboid, that one cuboid represent only one subtype of object in each category. We ablate the important modules of our model in Section 4.4.

heading: 4.1 EXPERIMENTAL SETUP

text:
Evaluation. The task of 3D object pose estimation involves the prediction of three rotation parameters (azimuth, elevation, in-plane rotation) of an object relative to the camera. In our eval-
uation, we follow the protocol as proposed in related work Zhou et al. (2018) to measure the pose estimation error between the predicted rotation matrix and the ground truth rotation matrix: ∆ (Rpred, Rgt) = ‖logm(RTpredRgt)‖F√
2 . We report two commonly used evaluation metrics the me-
dian of the rotation error, the percentage of predicted angles within a given accuracy threshold. Specifically, we use the thresholds pi6 and pi 18 . Following Zhou et al. (2018), we assume the centers and scales of the objects are given in all experiments.
Datasets. We evaluate NeMo on both the PASCAL3D+ dataset Xiang et al. (2014) and the occluded PASCAL3D+ dataset Wang et al. (2020). PASCAL3D+ contains 12 man-made object categories with 3D pose annotations and 3D meshes for each category respectively. We follow Wang et al. (2020) and Bai et al. (2020) to split the PASCAL3D+ into a training set with 11045 images and validation set with 10812 images. The occluded PASCAL3D+ dataset is a benchmark to evaluate robustness under occlusion. This dataset simulates realistic man-made occlusion by artificially superimposing occluders collected from the MS-COCO dataset Lin et al. (2014) on objects in PASCAL3D+. The dataset contains all 12 classes of objects from PASCAL3D+ dataset with three levels of occlusion, where L1: 20-40%, L2: 40-60%, L3: 60-80% of the object area are occluded.
We further test NeMo on the ObjectNet3D dataset Xiang et al. (2016), which is also a category-level 3D pose estimation benchmark. ObjectNet3D contains 100 different categories with 3D meshes, it contains totally 17101 training samples and 19604 testing samples, including 3556 occluded or truncated testing samples. Following Zhou et al. (2018), we report pose estimation results on 18 categories. Note that different from StarMap, we use all images during evaluation, including occluded or truncated samples.
Training Setup. In the training process, we use the 3D meshes (see Section 4.2 for experiments without the mesh geometry), the locations and scales of objects, and the 3D poses. We use Blender Community (2018) to reduce the resolution of the mesh because the meshes provided in PASCAL3D+ have a very high number of vertices. In order to balance the performance and computational cost, in particular the cost of the rendering process, we limit the size of the feature map produced by backbone to 18 of the input image. To achieve this, we use the ResNet50 with two additional upsample layers as our backbone. We train a backbone for each category separately, and learn a Neural Mesh Model for each subtype in a category. We follow hyperparameter settings from Bai et al. (2020) for the contrastive loss. We train NeMo for 800 training epochs with a batch size of 108, which takes around 3 to 5 hours to train a category using 6 NVIDIA RTX Titan GPUs.
Baselines. We compare our model to StarMap Zhou et al. (2018) using their official implementation and training setup. Following common practice, we also evaluate a popular baseline that formu-
lates pose estimation as a classification problem. In particular, we evaluate the performance of a deep neural network classifier that uses the same backbone as NeMo. We train a category specific Resnet50 (Res50-Specific), which formulates the pose estimation in each category as an individual classification problem. Furthermore, we train a non-specific Resnet50 (Res50-General), which performs pose estimation for all categories in a single classification task. We report the result of both architectures using the implementation provided by Zhou et al. (2018).
Inference via Feature-level Rendering. We implement the NeMo inference pipeline (see 3.4) using PyTorch3D Ravi et al. (2020). Specifically, we render the Neural Mesh Models into feature map F̄ using the feature representations Θ stored at each mesh vertex. We estimate the object pose by minimizing the reconstruction loss as introduced in Equation 12. For initialization of pose optimization, we uniformly sample 144 different poses (12 azimuth angles, 4 elevation angles, 3 in-plane rotations respectively). Then we pick the initial pose with minimum reconstruction loss as a starting point of optimization (optimization conduct with only the chosen pose, others will be deprecated). On average each image takes about 8s with a single GPU for inference. The whole inference process on PASCAL3D+ takes about 3 hours using a 8 GPU machine.

heading: 4.2 ROBUST 3D POSE ESTIMATION UNDER OCCLUSION

text:
Baseline performances. Table 1 (for categories specific scores, see 6) illustrates the 3D pose estimation results on PASCAL3D+ under different levels of occlusion. In the low accuracy setting (ACCπ
6 ) StarMap performs exceptionally well when the object is non-occluded (L0). However,
with increasing level of partial occlusion, the performance of StarMap degrades massively, falling even below the basic classification models Res50-General and Res50-Specific. These results highlight that today’s most common deep networks for 3D pose estimation are not robust. Similar, generalization patterns can be observed for the high accuracy setting (ACC π
18 ). However, we can
observe that the classification baselines do not perform as well as before, and hence are not well suited for fine-grained 3D pose estimation. Nevertheless, they outperform StarMap at high occlusion levels (L2 & L3).
NeMo. We evaluate NeMo in three different setups: NeMo uses a down-sampled object mesh as geometry representation, NeMo-MultiCuboid and NeMo-SingleCuboid approximate the 3D object geometry crudely using 3D cuboid boxes. We discuss the cuboid generation and results in detail in the next paragraph. Compared to the baseline performances, we observe that NeMo achieves competitive performance at estimating the 3D pose of non-occluded objects. Moreover, NeMo is much more robust compared to all baseline approaches. In particular, we observe that NeMo achieves the highest performance at every evaluation metric when the objects are partially occluded. Note that the training data for all models is exactly the same. To further investigate and understand the robustness of NeMo, we qualitatively analyze the pose estimation and occluder location predictions of NeMo in Figure 3. Each subfigure shows the input image, the pose estimation result, the occluder localization map and the loss as a function of the pose angles. We visualize the loss landscape along each pose parameter (azimuth, elevation and in-plane rotation) by sampling the individual parameters in a fixed step size, while keeping all other parameters at their ground-truth value. We further split the binary occlusion mapZ into three regions to highlight the occluder localization performance of NeMo. In particular, we split the region that is explained by the background model into a yellow and a red region. The red region is covered by rendered mesh and highlights the locations with the projected region of the mesh, which the neural
mesh model cannot explain well. Hence these mark the locations in the image that NeMo predicts to be occluded. From the qualitative illustrations, we observe that NeMo maintains high robustness even under extreme occlusion, when only a small part of the object is visible. Furthermore, we can clearly see that NeMo can approximately localize the occluders. This occluder localization property of NeMo makes our model not just more robust but also much more human-interpretable compared to standard deep network approaches.
NeMo without detailed object mesh. We approximate the object geometry in NeMo by replacing the downsampled mesh with 3D cuboid boxes (see Figure 5). The vertices of the cuboid meshes are evenly distributed on all six sides of the cuboid. For generating the cuboids, we use three constraint: 1) The cuboid should cover all the vertices of the original mesh with minimum volume; 2) The distances between each pair of adjacent vertices should be similar; 3) The total number of vertices for each mesh should be around 1200 vertices. We generate two different types of models. NeMo-MultiCuboid uses a separate cuboid for each object mesh in an object category, while NeMoSingleCuboid uses on cuboid for all instances of a category. We report the pose estimations results with NeMo using cuboid meshes in Table 1. The results show that approximating the detailed mesh representations of a category with a single 3D cuboid gives surprisingly good results. In particular, NeMo-SingleCuboid even often outperforms our standard model. This shows that generative models of neural network feature activations must not retain the detailed object geometry, because the feature activations are invariant to detailed shape properties. Moreover, NeMo-MultiCube outperforms the SingleCube model significantly. This suggests that for some categories the size between different sub-types can be very differnt (e.g. for the airplane class it could be a passanger jet or a fighter jet). Therefore, a single mesh may not be representative enough for some object categories. The MultiCuboid model even outperforms our the model with detailed mesh geometry. This is very likely caused by difficulties during the down-sampling of the original meshes in PASCAL3D+, which might remove important parts of the object geometry. We also conduct experiment on ObjectNet3D dataset, which reported in Table 2. The result demonstrates that NeMo outperforms StarMap in 14 categories out of all 18 categories. Note that due to the considerable number of occluded and truncated images in ObjectNet3D dataset, this dataset is significantly harder than PASCAL3D+, however, NeMo still demonstrates reasonable accuracy.

heading: 4.3 GENERALIZATION TO UNSEEN VIEWS

text:
To further investigate robustness of NeMo to out-of-distribution data, we evaluate the performance of NeMo when objects are observed from previously unseen viewpoints. For this, we split the PASCAL3D+ dataset into two sets based on the ground-truth azimuth angle. In particular, we use
the front and rear views for training. We evaluate all approaches on the full testing set and split the performance into seen (front and rear) and unseen (side) poses. The histogram on the left of Table 4 shows the distribution of ground-truth azimuth angles in the PASCAL3D+ test dataset. The seentest-set contains 7305 images while the unseen-test-set contains 3507 images. Table 4 shows that NeMo can significantly better generalize to novel viewpoints compared to the baselines. For some categories the accuracy of NeMo on the unseen-test-set is even comparable to seen-test-set (Table 7). These results highlight the importance of building neural networks with 3D internal representations, which enable them to generalize exceptionally well to unseen 3D transformations.

heading: 4.4 ABLATION STUDY

text: In Table 4, we study the effect of each individual module of NeMo. Specifically, we remove the clutter feature, background score and occluder prediction during inference, and only use foreground score to calculate pose loss. This reduces the robustness to occlusion significantly. Furthermore, we remove the contrastive loss and use neural features that were extracted with an ImageNet-pretrained Resnet50 with non-parametric-upsampling. This leads to a massive decrease in performance, and hence highlights the importance of learning locally distinct feature representations. Table 5 (and Table 10) study the sensitivity of NeMo to the random pose initialization before the pose optimization. In this ablation, we evaluate NeMo-MultiCuiboid with 144 down to 1 uniformly sampled initialization poses. Note that we do not run 144 optimization processes. We instead evaluate the reconstruction error for each initialization and start the optimization from the initializaiton with the lowest error. Hence, every experiment only involves one optimization run. The results demonstrate that NeMo benefits from the smooth lose landscape. With 6 initial samples NeMo achieves a reasonable performance, while 72 initial poses almost yield the maximum performance. This ablation clearly highlights that, unlike standard Render-and-Compare approaches Blanz & Vetter (1999); Schönborn et al. (2017), NeMo does not require complex designed initialization strategies.

heading: 5 CONCLUSION

text: In this work, we considered the problem of robust 3D pose estimation with neural networks. We found that standard deep learning approaches do not give robust predictions when objects are partially occluded or viewed from an unseen pose. In an effort to resolve this fundamental limitation we developed Neural Mesh Models (NeMo), a neural network architecture that integrates a prototypical mesh representation with a generative model of neural features. We combine NeMo with contrastive learning and show that this makes possible to estimate the 3D pose with very high robustness to out-of-distribution data using simple gradient-based render-and-compare. Our experiments demonstrate the superiority of NeMo compared to related work on a range of challenging datasets.

heading: ACKNOWLEDGMENTS

text: We gratefully acknowledge funding support from ONR N00014-18-1-2119, ONR N00014-20-12206, the Institute for Assured Autonomy at JHU with Grant IAA 80052272, and the Swiss National Science Foundation with Grant P2BSP2.181713. We also thank Weichao Qiu, Qing Liu, Yutong Bai and Jiteng Mu for suggestions on our paper.

heading: A APPENDIX

title: NEMO: NEURAL MESH MODELS OF CONTRASTIVE FEATURES FOR ROBUST 3D POSE ESTIMATION