We propose a guided dropout regularizer for deep networks based on the evidence of a network prediction: the firing of neurons in specific paths. In this work, we utilize the evidence at each neuron to determine the probability of dropout, rather than dropping out neurons uniformly at random as in standard dropout. In essence, we dropout with higher probability those neurons which contribute more to decision making at training time. This approach penalizes high saliency neurons that are most relevant for model prediction, i.e. those having stronger evidence. By dropping such high-saliency neurons, the network is forced to learn alternative paths in order to maintain loss minimization, resulting in a plasticity-like behavior, a characteristic of human brains too. We demonstrate better generalization ability, an increased utilization of network neurons, and a higher resilience to network compression using several metrics over four image/video recognition benchmarks.
The authors propose a data-dependent dropout variant that produces dropout candidates based on their predictive saliency / relevance. Results are reported for 4 datasets (Cifar10, Cifar100, Caltech256 and UCF101) and 4 different models (CNN-2, AlexNet, VGG16 and VGG19), and suggest an increase in generalization performance over other dropout approaches (curriculum dropout, standard dropout, no dropout), as well as increase in the network's plasticity as measured by some existing metrics from the literature. The authors conclude that Excitation Dropout results in better network utilization and offers advantages for network compression (in the sense of neuron pruning).
Overall I find the idea to be interesting and fairly novel, and commend the authors for the fluid writing style. However, I find key issues with the testing and experiments. Specifically, the lack of confidence bounds for individual results makes it impossible to determine whether the reported incremental improvements are actually significant over those of existing approaches. Likewise, I criticize the choice of methods the authors have chosen to compare against, as several other data-dependent dropout approaches (e.g. Information Dropout) exist that may be conceptually closer (and therefore more comparable) to the proposed approach. I also question the choice of tested network architectures and the placement of the dropout layer.
The paper could be of high significance if all claims in the paper could be backed up by experiments that show the advantage of Excitation Dropout to be not a random effect. I will therefore give a lower score for the paper in its current form, but am willing to revise my rating the major points below are properly addressed.
Pros:
+ novel mechanism to improve dropout, results seemingly superior over other methods
+ achieves better utilization of network resources and achieves robustness to dropping out neurons at test time
Cons:
- results without error bars, unclear if advantage is significant
- did not compare against most relevant competing methods
MAJOR POINTS
Section 2 - The comparison to Moreiro et al. is not entirely clear. A fairer comparison would be with some of the other methods listed which also focus on answering the question of which neurons to dropout, or approaches which determine the dropout policy based on information gained from the data, such as Information Dropout (Achille & Soatto). The authors state that Morerio et al are the state-of-the-art in dropout techniques, however based on the results presented here (Figure 3) it seems to perform just as well as standard dropout. Perhaps there are architecture-specific or data-specific issues? In any case this example undermines the confidence of the claims.
Section 3.2, equation 3 - is there some theoretical underpinning as to how this equation was modelled, or was it chosen simply because it covers the expected corner cases described in paragraph 4 of this section? Also, given the intuition in this paragraph (e.g. p_EB = 1 / N), it is correct to assume this equation models the dropout probability but only for fully connected layers? What about dropout in convolutional layers? Though some previous statements do point to the usage of dropout predominantly for fully connected layers, I feel that this context is missing here and should be explicitly addressed. The caption to e.g. Table 1 seems to imply the authors add a single dropout layer in one of the fully connected layers, however this begs the question as to why this positioning was chosen - why only one dropout layer, and why precisely at that location? The scope of the claims should be adapted accordingly.
Section 4.2 - "After convergence, ED demonstrates a significant improvement in performance compared to other methods". If five trained models were used, then some sense of measure of uncertainty should be given throughout. For example, in the Cifar10 results for Figure 3, it is difficult to say whether the marginal improvement from about 80% (standard dropout and curriculum dropout) to about 82% (excitation dropout) is significant or not. Perhaps this would be less of an issue if the authors had worked with e.g. ImageNet, but for these smaller datasets it would definitely be worth to be on the safe side. I highly suspect that statistically speaking (perhaps with the exception of the results on Caltech256), the effects of all of these dropout variants are indistinguishable from each other. I urge the authors to include a measure of the standard deviation / 95% confidence interval across the models that were tested.
The results presented sub-section 4.3 do not justify the claim that the models trained with Excitation Dropout tend to be more informative. Perhaps the definition of "informative" should be expanded upon in length. Can the authors show that the alternative paths learned by the models augmented with Excitation Dropout indeed carry complimentary information and not just redundant information?
Figure 5 shows interesting results, but once again begs the question of whether there is any significant difference between standard dropout and curriculum dropout. I encourage the authors to include confidence bounds for each trace. Likewise, there is an inherent bias in the results, in that the leftmost figure compares EB and CD in the context in which EB was trained, i.e. dropping of "most salient" neurons. The comparison is one-sided, however, as no results are reported from the context in which CD was trained, i.e. dropping neurons more frequently as training progresses. Comparing these results would bring to light whether the performance boost see in Figure 5 is a function of "overfitting" to the training manner or not.
Also, I believe the results for the second column (dropping out least relevant neurons) are misleading. To the best of my understanding, as p_c increases, at some point neurons start to be dropped that actually have high relevance. This could explain why all curves start out similarly and EB slowly begins to stick out - at this point the EB models once again start to be used in the context within which they were trained, in contrast to the other approaches. The authors should perhaps also explicity clarify why this second column gives any more information than the first.
MINOR POINTS
The authors propose Excitation Dropout as a guided regularization technique. Batch normalization is another standard regularization technique which is often compared with dropout. In particular, for deep CNNs, batch normalization is known to work very well, often better than the standard dropout. In the experiments here, to what extent was batch normalization and / or any other widely utilized network regularizers used? Is it possible that the regularizing effect found here actually comes from one of these? I.e. were the models that were not trained from scratch trained with batch normalization? It would be good if more data could be provided for EB vs. other regularizing techniques, if the claim is that EB is a novel regularizer.
Section 3.1 - "We choose to use EB since it produces a valid probability distribution for each network layer". Though this is a nice property, were there any other considerations for choosing the saliency method? Recent work (Adebayo et al, "Sanity Checks for Saliency Maps") has shown that even some well established saliency techniques are actually independent from both the model and data. As this approach relies heavily on the correctness of EB, I feel that a further justification should be given to validate its use for this scenario other than just based on the type of output it produces.
Section 3.1, equation 2 - more detail and reasoning should be given as to why connections with negative weights are excluded from the computation of the conditional probability, if possible without referring the reader to the EB paper. Why is this justified? Is this probability modelled for a specific activation function? 
The authors do not provide the details of the CNN-2 architecture (even in the appendix) and simply refer to another article. If the majority of the results presented in the paper are based on this network (including a reference made to a specific layer of the network in subsection 4.2) – which is not commonly known – why not to detail the network architecture and save additional effort for the reader?
How are the class-wise training and test images chosen for Caltech256 dataset?
The authors test the CNN-2 architecture on Cifar10 and Cifar100, and AlexNet, VGG16, and VGG19 on UCF101. I feel that at least a couple architectures should be validated with more than a single dataset, or the authors should justify the current matching between architectures and datasets. Table 2 is unclear regarding what models were used for what datasets (caption could be interpreted to mean that VGG16 was also used for Cifar and Caltech, however other statements seem to say otherwise).
"To prove that the actual boost in accuracy with ED is not provided by the choice of specific masks,..." I suggest that the authors rephrase or explain this sentence in more detail. To the best of my understanding, it is precisely the fact that different masks are used, each reflective of the particular input used to generate the forward activations, that gives boost in performance over "standard" dropout methods by identifying salient paths in the network.
Although it is a very important experimental detail, only in the end of sub-section 4.2, it becomes clear in which layers Excitation Dropout was applied. 
Y-axis labels are missing for the left panels in Figure 3.
The authors randomly choose to abbreviate Excitation Dropout as ED in some paragraphs, while write the full form in others.  
Table 2 - It is not clear that the "Neurons ON" metric refers to the "average percentage of zero activations" explained below. 
Table 2 - How is peak p_EB measured? Is this an average over a set of test images after convergence? If so, I similarly suggest for confidence bounds to be introduced. It would be interesting to compare this to intermediate values (e.g. after every epoch) during training. Same question for entropy of activations and entropy of pEB. This information would be useful for reproducibility.
Table 2 - Where do the delta values in Table 2 come from? If empirically determined, it should be stated explicitly.
Table 2 - In general, because the metrics provided in Table 2 are averages (second paragraph of this Section 4.3), both (to the best of my understanding) across input subsets (e.g. averging results over many test inputs) and models (caption to Table 1), I feel Table 2 in its current form raises confusion given the lack of confidence bounds. I recommend the authors to clarify what type of averaging was done and to introduce e.g. standard deviations across all reported scores. The authors should refrain from using the term "significantly" while describing results if no statistical testing was done, or explicitly clarify their usage of this term.
Table 2 - In general, Table 2 reports results on selected metrics which, if the authors' hypothesis is correct, should have a clear trend as training progresses. An interesting idea to explore would be to include an analysis (in the appendix) of how these factors change over the course of the training procedure. Intuitively, it seems plasticity is something that should be learned slowly over time, so if these plots were to reveal something different, it would be indicative that something else is going on.
Figure 4 - Judging heatmaps is difficult as it depends on the visual perception of the reader. Thus, it is difficult to judge whether, as the authors claim, ED is indeed less "peaky" than the other alternatives. I suggest that the authors use a perceptually uniform heatmap, and to acompany these figures with e.g. the histogram of the heatmap values. Likewise, it is unclear how the multi-model aspect of the testing plays a role in generating these results. From the 5 originally trained models, how was the model selected that generated these results? Was there averaging of any kind?
Figure 5: the text is too small to be readble
Is "re-wiring" the most appropriate term to use to describe what is happening at inference time? Although different paths may be used, the network connections themselves are fixed and thus this is a potential source for confusion.
What do numbers in Table-4 in the appendix represent? Test accuracy?