While variational dropout approaches have been shown to be effective for network sparsification, they are still suboptimal in the sense that they set the dropout rate for each neuron without consideration of the input data. With such input-independent dropout, each neuron is evolved to be generic across inputs, which makes it difficult to sparsify networks without accuracy loss. To overcome this limitation, we propose adaptive variational dropout whose probabilities are drawn from sparsity-inducing beta-Bernoulli prior. It allows each neuron to be evolved either to be generic or specific for certain inputs, or dropped altogether. Such input-adaptive sparsityinducing dropout allows the resulting network to tolerate larger degree of sparsity without losing its expressive power by removing redundancies among features. We validate our dependent variational beta-Bernoulli dropout on multiple public datasets, on which it obtains significantly more compact networks than baseline methods, with consistent accuracy improvements over the base networks.
The paper proposes Variational Beta-Bernoulli Dropout to sparsify the parameters of the network. The authors also proposed an input-dependent dropout using input-dependent Beta-Bernoulli priors. The paper presents necessary theoretical details as well as the experimental comparison with the other dropout methods on MNIST and CIFAR 10/100 datasets.
The paper is well-written overall and easy to follow. The paper provides a thorough background on previous work; however the motivation for having an input-dependent dropout method is relatively weak. 
I believe the paper presents main idea and necessary details thoroughly, though I have some confusions:
- In sections 4.1 and 4.2 (more specifically formula 7, 8 and 18), is there a global z (i.e. only one z), or there is a separate z for each layer? I implicitly inferred it's per layer but it's not clear from the paper.
- The paper explains well how the two proposed dropouts can be learned during training, but it's not clear about the inference, specially in variational dependent Beta-Bernoulli dropout. In section 5, experimental details, it is said: "In the testing phase, we prune the neurons/filters whose expected dropout mask probability E_q[pi_k] are smaller than a fixed threshold 10^-3." This is not clear to me what it means, and also the related footnote which says different thresholds were tried but the difference is insignificant! More elaboration is helpful.
- In the experimental results, the authors reported median and standard-deviation. Is there any reason the authors didn't report mean and standard-deviation instead? Also, it is not clear if xFLOP and Memory report the best, mean or median of 3 runs.
- In Table 1, what does column Neurons represent? Is it the model parameters at inference time? and if yes, is it the smallest number of parameters in 3 runs? In Table 2, how large the network is in terms of model parameters?
- In the experimental results, some models are missing that are mentioned in the paper as previous work which seem to be good to use for comparison, such as Sparse Variational Dropout (Molchanov 2017) and Structured Sparsity Learning (Wen et al 2016).
- I encourage the authors to expand more on analyzing the experimental results! In Table 1 and Table 2, there are 2 lines for BB and DBB, which is helpful to explain what each line represents. Looking at the results of all tables, and comparing different dropouts w.r.t :1) Error, 2) xFLOPs and 3) Memory, I cannot draw a clear conclusion from the results. For example, in Table 2, CIFAR-10, comparing first line of BB to SBP and second line to VIB, the difference is not significant. More analysis about the results can be helpful!