Recently convolutional neural networks (CNNs) achieve great accuracy in visual recognition tasks. DenseNet becomes one of the most popular CNN models due to its effectiveness in feature-reuse. However, like other CNN models, DenseNets also face overfitting problem if not severer. Existing dropout method can be applied but not as effective due to the introduced nonlinear connections. In particular, the property of feature-reuse in DenseNet will be impeded, and the dropout effect will be weakened by the spatial correlation inside feature maps. To address these problems, we craft the design of a specialized dropout method from three aspects, dropout location, dropout granularity, and dropout probability. The insights attained here could potentially be applied as a general approach for boosting the accuracy of other CNN models with similar nonlinear connections. Experimental results show that DenseNets with our specialized dropout method yield better accuracy compared to vanilla DenseNet and state-of-the-art CNN models, and such accuracy boost increases with the model depth.
Overall Thoughts:
I think the use of regularisation to improve performance in DenseNet architectures is a topic of interest to the community. My concern with the paper in it’s current form is that the different dropout structures/schedules are priors and it is not clear from the current analysis exactly what prior is being specified and how to match that to a particular dataset. Further, I believe that the current presentation of the empirical results does not support the nature of the claims being made by the authors. I would be very interested to hear the authors’ comments on the following questions.
Specific Comments/Questions:
Sec1: Sorry if I have missed something but for the two reasons against std dropout on dense net, the reference supports the second claim but could a reference be provided to substantiate the first?
Sec1/2: The discussion around feature re-use needs to be clarified slightly in my opinion. Dropout can provide regularisation in a number of regimes - the term “feature reuse” is a little tricky because I can see the argument from both sides - under the authors arguments, forcing different features to be used can be a source or robustness so would not the level of granularity be something to be put in as a prior and not necessarily inherently correct or incorrect?
Sec3: The key contribution (in my opinion) suggested by the authors is the “detailed analysis” of their dropout structures. I’m afraid I didn’t see this in this section - there are a number of approaches that have been taken in the literature to analyse the regularisation properties of dropout - e.g. the insightful approach of Gal and Ghahramani on dropout as a Bayesian regulariser (as well as others). I was expecting to see something similar to this - could the authors comment on this? Would such an analysis be possible - it would reveal the true priors being applied by the different approaches and allow an analysis of the priors being applied by the different methods?
Sec3: Similarly, with the dropout probability schedules, there are practical methods for learning such probabilities during training (e.g. Concrete Dropout) - would it not be possible to learn these parameters with these approaches? Why do we need to set them according to fixed schedules? I think it would be necessary to demonstrate that a fixed schedule outperforms learned parameters.
Sec4: My main difficulty here is that the other key contribution of the paper are the claims constructed around empirical results. Throughout the results section, only single values are presented without attempt to measure the distributions of the results (not even error bars). Without this information it is impossible to make any statements on the significance of the results. Ideally histograms should be provided (rather than just error bars). How do we know the changes conferred are significant for the particular problems? How do we know that they are causal from the new structures and not from hyper parameters or optimisation effects?
Sec4: Dropout is the application of a prior - how do we know what this prior is doing and when it is sensible to apply it? How do we know the results will transfer to datasets other than CIFAR?
Sec4: Please could the authors provide justification to the claim that the improvements would increase with the depth of the network?
Refs: Please could the authors be sure to cite the published versions of articles (not ArXiv versions) when papers have been peer reviewed - e.g. the citation for DenseNet (among others)
Other Points:
Could the authors use text mode for sub or superscripts in maths equations when using words as opposed to symbols?
There are a number of uses of “could” when I don’t think the authors mean “could” - please could this be checked?
Typos:
p4 replying -> relying, whcih -> which