Deep neural networks (DNNs) build high-level intelligence on low-level raw features. Understanding of this high-level intelligence can be enabled by deciphering the concepts they base their decisions on, as human-level thinking. In this paper, we study concept-based explainability for DNNs in a systematic framework. First, we define the notion of completeness, which quantifies how sufficient a particular set of concepts is in explaining a model's prediction behavior. Based on performance and variability motivations, we propose two definitions to quantify completeness. under degenerate conditions, our method is equivalent to Principal Component Analysis. Next, we propose a concept discovery method that considers two additional constraints to encourage the interpretability of the discovered concepts. We use game-theoretic notions to aggregate over sets to define an importance score for each discovered concept, which we call ConceptSHAP. On specifically-designed synthetic datasets and real-world text and image datasets, we validate the effectiveness of our framework in finding concepts that are complete in explaining the decision, and interpretable.
Summary 
The paper proposes metrics for evaluating concept based explanations in terms of ‘completeness’ -- characterized by (1) whether the set of presented concepts if sufficient to retain the predictive performance of the original model and (2) how is performance affected when all information useful to a complete set of concepts (as per (1)) is removed from features at a specific layer. Assuming concept vectors lie in linear sub-spaces of the activations of the network at a specific layer, the underlying assumption is that if a given set of ‘concept’ vectors is complete, then using a projection of the intermediate features from input onto the sub-space spanned by concepts should not result in reduced / affected predictive performance. Based on these characterizations, the paper proposes an objective to discovering complete and interpretable set of concepts given a candidate cluster of concepts. Furthermore, the paper proposes metrics to quantify the importance of each concept (using Shapley values) and per-class importance of concepts. The authors conduct experiments on toy data, image and text classification datasets and show that their proposed approach can discover concepts that are complete and interpretable.
Strengths
- The paper is well-written and generally easy to follow. The authors do a good job of motivating the need for the completeness metric, characterizing it specifically in the case of concepts spanning sub-spaces of activations and subsequently utilizing the same to motivate an effective concept discovery method.
- The proposed approach and angle being looked at in the paper is novel in the sense that while prior work has mostly focused on characterizing concepts which are salient. Ensuring that concepts are sufficient for predictive performance ensures the fidelity of the interpretability approach.
- I like the fact that the authors decided to capture both aspects of the completeness criterion -- (1) projection to concept-space should not hurt performance and (2) how does removing concept-projected information from the features affect performance. Capturing both provides a holistic viewpoint of the features of the concerned layer -- (1) can explain features/decisions with an associated metric based on the ‘imperfect’ set of concepts and (2) captures the effectiveness of the information present in the features if we remove all concept-useful information.
- The choice of using Shapley values in ConceptSHAP as a metrics provides a whole range of desirable properties in the metrics used indicate the quality of concepts at different levels of granularity -- per-class importance of concepts across classes adding up to the overall importance of a concept. Furthermore, the observation that under certain conditions, the top-k PCA vectors maximize the defined completeness scores is interesting as well.
Weaknesses
Having said that, I’m interested to hear the thoughts of the authors on the below two points. My primary (not major) concern is the fact the proposed approach is only centered around ensuring and evaluating fidelity of the discovered concepts to the original model. 
- While the proposed approach and evaluation metrics are novel, and the results generally support the claims of the paper -- toy experiments result in recovery of the ground-truth concepts, concepts identified for text and image classification offer feasible takeaways -- there is still a lack of proper human-interpretability aspect of the discovered concepts. The proposed approach to discover complete concepts mostly acts similar to a pruning approach on top of a candidate set of concepts based solely on fidelity to the original model. The obtained concepts are mostly explained via feasible hypotheses. One possible experiment that can be used to capture the reliability aspect of the discovered concepts could be as follows -- “Given the set of concepts (and representative patches) and SHAP values across all (or most relevant) classes, are humans able to predict the output of the model?” Is it possible to setup and experiment of this sort? I believe it might help understand the utility of the SHAP values in this context (beyond the advantages in terms of manipulation and characterized completeness).
- The experimental results for image classification are presented on the Animals with Attributes (AwA) dataset. AWA is a fine-grained dataset with only one class present per-image. I’m curious to what happens when the same approach is applied to datasets where images have multiple classes (and potentially distractor classes) present. Is it possible that it becomes harder to discover ‘complete’ concepts (subject to the availability of a decent approach to provide an initial set of candidate clusters). Do the authors have any thoughts on this and any potential experiments that might address this?
Reasons for rating
Beyond the above points of discussion, I don’t have major weaknesses to point out. I generally like the paper. The authors do a good job of identifying the sliver in which they make their contribution and motivate the same appropriately. The proposed evaluation metrics and discovery objectives offer several advantages and can therefore generally serve as useful quantifiers of concept-based explanation approaches. The strengths and weaknesses highlighted above form the basis of my rating.