There has long been debates on how we could interpret neural networks and understand the decisions our models make. Specifically, why deep neural networks tend to be error-prone when dealing with samples that output low softmax scores. We present an efficient approach to measure the confidence of decision-making steps by statistically investigating each unit's contribution to that decision. Instead of focusing on how the models react on datasets, we study the datasets themselves given a pre-trained model. Our approach is capable of assigning a score to each sample within a dataset that measures the frequency of occurrence of that sample's chain of activation. We demonstrate with experiments that our method could select useful samples to improve deep neural networks in a semi-supervised leaning setting.
Unlike traditional supervised learning that requires fully labeled data to train a classifier, semisupervised learning requires only a portion of data to be labeled, which demands less effort in data preparation and is more practical in real life. Consider a semi-supervised leaning task where both the labeled and the unlabeled data come from the same sampling distribution. A straightforward technique is that we first train a model on the labeled data, and then use this model to classify the unlabeled parts of the dataset to enlarge the training set. After this, all the data are labeled regardless of correctness. The model could then be re-trained by iteratively performing the above steps. However, experiments have shown that this approach does not work well on CNNs (see section 3 for details). The reason is two-fold:
• Parts of the data are labeled by our model correctly with extreme confidence, i.e. probability generated by softmax layer is close to 1. This could happen only if our model already learned the features during training, therefore adding these data into the training set will not provide any improvement.
• The model does not pick up some of the features that are contained in the unlabeled data during training. Therefore, those data are doomed to be labeled incorrectly and could potentially weaken our model upon added into the training set.
To summarize these two problems: could we use a particular sample to improve our model; and could we trust our model to label a particular sample correctly. We propose a method to automatically assign scores to data that represents both confidence (could we trust it) and interpretability (could we use it) based on a pre-trained network. Our approach mainly focuses on activations of individual neurons within a network. For a pre-trained CNN and some images within one category, we focus on images' activation maps generated by all of the convolutional and linear layers. The activation maps contain information about each image's relationship with the activated neurons. Then our method could identify neurons that are commonly activated and thereby assigning scores to each individual image based on it's chain of activation. Samples with high scores are assumed to be interpretable and experiments have shown that they are also trustworthy. This entire procedure is unsupervised, i.e. our approach does not need label information to assign image scores. Therefore, we could apply our method to semi-supervised learning tasks including image classification and object detection.
Structure of this paper is the following: in section 2 we formally define image scores and then examine some key properties of our approach; in section 3 we conduct experiments in a semisupervised learning setting to demonstrate how scoring the images could help with improving a pre-trained model, experiments are done in image classification and object detection tasks; section 4 concludes the entire work.
This paper proposes a new metric called the image score that compares the similarity of activation between a given image with a pool of groundtruth images. The paper finds it useful for semi-supervised learning with self-teaching, where the network picks the most confident sample and use the network prediction as the label. It finds that the proposed method is better than 1) not using the unlabeled data and 2) using softmax as an indicator for model prediction certainty.
Motivation: The introduction begins by motivating the interpretability story of deep learning, but I don’t see gaining any more interpretability by reading the rest of the paper. The paper proposes to improve interpretability by assigning a score to each individual example, but then the obtained scores are not properly analyzed in the paper, and only final classification accuracy is evaluated. What are the training samples that makes the model make certain decision at test time? How to measure the correlation between the usefulness of training samples and the proposed image score? These questions left unanswered in the paper. Figure 1 helps a little bit, but then the top row is not necessarily the bad images, but maybe hard examples that needs extra attention to learn. Therefore, I think the end results presented in the experiments do not align with the motivation. Rather than shooting for interpretability, this is just another semi-supervised learning paper.
Models: The major issue of this paper is the model formulation that is not well motivated. The intuition of how the authors come up with the equation for computing the image score is not well explained. Hence the formulation seems very ad-hoc, and it is unclear why this is the selected method.
Experiments: As a semi-supervised learning paper, a common setting for CIFAR-10 is to use 4k labeled images. Here, the method uses 30k, which is 7.5x the size of the usual setting. It also does not compare to prior semi-supervised learning work (e.g. one of the recent one is: https://arxiv.org/abs/1711.00258). The only two baselines discussed here are weak. Also the improvement from the baselines by using the proposed method is not very significant.
Comparison: Figure 2-4 shows some positive correlation between the accuracy and score, which is fair, but it doesn’t compare to any baselines--the only one we have is softmax baseline and it is not shown in the figure.
In conclusion, I couldn’t see how the paper improves interpretability as claimed in the introduction. The proposed method seems ad-hoc, without any justification. Being considered as a semi-supervised learning paper, it lack significant amount of comparison to prior work and adopting a common semi-supervised benchmark. Due to the above reasons, I recommend reject.
---
Minor points:
“...almost all of the existed works investigate only the models and ignore the relationship between models and samples”. This is over-exaggerated. I believe most of the visualization techniques are dependent on the actual input samples. It is true to say about “training samples” not “samples” in general.
“all correctly classified images should have similar chain of activation, while incorrectly classified images should have very different activations both within themselves and with correctly classified images”. This claim seems not backed up. How do you know it is the case for “all” correctly classified images? What defines similar/different?