Convolutional neural networks (CNNs) have often been treated as "black-box" and successfully used in a range of tasks. However, CNNs still suffer from the problem of filter ambiguity -an intricate many-to-many correspondence relationship between filters and features, which undermines the models' interpretability. To interpret CNNs, most existing works attempt to interpret a pre-trained model, while neglecting to reduce the filter ambiguity hidden behind. To this end, we propose a simple but effective strategy for training interpretable CNNs. Specifically, we propose a novel Label Sensitive Gate (LSG) structure to enable the model to learn disentangled filters in a supervised manner, in which redundant channels experience a periodical shutdown as flowing through a learnable gate varying with input labels. To reduce redundant filters during training, LSG is constrained with a sparsity regularization. In this way, such training strategy imposes each filter's attention to just one or few classes, namely class-specific. Extensive experiments demonstrate the fabulous performance of our method in generating sparse and highly class-related representation of the input. Moreover, comparing to the standard training strategy, our model displays less redundancy and stronger interpretability.
Contributions: The paper proposes a novel Label Sensitive Gate (LSG) structure to enable the model to learn disentangled filters in a supervised manner. The novelty of the paper is to introduce the Label-Sensitive Gate path during the training, on top of the standard training path. This encourages the filters to extract class-sensitive features. 
The highlight of the experimental results is that by incorporating the LSG structure, the author is able to achieve better localization with CAM because the filters are trained to extract more relevant features. Additionally, filters from different classes are kind of "orthogonal" to each other.
However, I have a few questions for the author:
a) In Figure 5, the author mentions "TP/FN" samples in the caption. However, for the plot at the right, it is titled with "False Positive". Is this a typo or do you talk about something else? 
b) In Figure 7, the author shows the correlation matrix of filters. Since the filters belonging to one class are highly correlated (block matrices in plot d), why do not train even less filters for each class? If we train less filters for each class, will that hurt the performance? It might be a good direction on how to interpret filters from one class and how to make intra-class filters more interpretable.
Overall, the novelty of the paper is limited by the scope of the experimental results. Why would people care about interpretability of CNN filters is not explained clearly. The paper shows interpretable filters could help with localization. However, this is the only example in the paper I find useful. It would be better if the author could show any other applications. Moreover, I wish the author could spend more paragraph discussing why filters belonging to different classes tend to be orthogonal to each other. This is not clearly written (Eq(3)?). A more theoretical/intuitive explanation would also make the paper stronger as well.
If the paper could omtroduce more real applications of the proposed LSG model and give more theoretical/intuitive explanations of why the inter-class filters are orthogonal, I would be willing to raise up my scores.