DropBlock: A regularization method for convolutional networks
Deep neural networks often work well when they are over-parameterized and trained with a massive amount of noise and regularization, such as weight decay and dropout. Although dropout is widely used as a regularization technique for fully connected layers, it is often less effective for convolutional layers. This lack of success of dropout for convolutional layers is perhaps due to the fact that activation units in convolutional layers are spatially correlated so information can still flow through convolutional networks despite dropout. Thus a structured form of dropout is needed to regularize convolutional networks. In this paper, we introduce DropBlock, a form of structured dropout, where units in a contiguous region of a feature map are dropped together. We found that applying DropbBlock in skip connections in addition to the convolution layers increases the accuracy. Also, gradually increasing number of dropped units during training leads to better accuracy and more robust to hyperparameter choices. Extensive experiments show that DropBlock works better than dropout in regularizing convolutional networks. On ImageNet classification, ResNet-50 architecture with DropBlock achieves 78.13% accuracy, which is more than 1.6% improvement on the baseline. On COCO detection, DropBlock improves Average Precision of RetinaNet from 36.8% to 38.4%.
(a) (b) (c) Figure 1: (a) input image to a convolutional neural network. The green regions in (b) and (c) include the activation units which contain semantic information in the input image. Dropping out activations at random is not effective in removing semantic information because nearby activations contain closely related information. Instead, dropping continuous regions can remove certain semantic information (e.g., head or feet) and consequently enforcing remaining units to learn features for classifying input image.
The authors present a version of dropout that zeros out square blocks of neurons rather than individual neurons. They experience robust improvements relative to dropout.  This is a good paper, with potentially very wide-spread applicability.   The authors only present validation error in all their studies, but not test error. They claim that this is "common practice". I quickly checked the ResNet / DenseNet / Stochastic Depth / ShakeShake and ShakeDrop papers. Only one paper (DenseNet) seems to use validation error consistently and one (ResNet) partially. So I wouldn't call reporting the validation error common practice. In any case, this is a terrible practice for obvious reasons so I would urge any author (including the authors of this paper) to break the trend by reporting test error. If the authors feel that reporting validation error is necessary for comparion, they can report both.    Did you use the validation set for hyperparameter tuning? If so, in what way? Since you write "we first estimate the block_size_factor to use (typically 0.5)", it seems you are using it for at least some tuning. How is reporting validation error on the final model then still unbiased?  I am confused by your treatment of the DropBlock hyperparameters. On page 3, you start by saying that you don't set gamma and block_size directly, but block_size_factor and keep_prob. But then in line 156 you say "gamma = 0.4". I am assuming you mean "keep_prob = 0.4"? Similarly, in table 2 / line 210, you use gamma, but I think you mean keep_prob.  How do you compensate for DropBlock? E.g. in dropout, if a feature is dropped with probability gamma, all other outputs are multiplied with 1/(1-gamma). How do you handle this in DropBlock?  Have you tried block_size_factor=1, i.e. dropping entire channels? What happens then?  ### Additional comments post-rebuttal ###  I apologize for the stupid comment regarding the imagenet test set. I didn't know that it is "secret".   "We are happy to create a holdout partition of training data and re-do the experiments for the final submission of 33 the paper." I am envious of your seemingly unlimited GPU access :) But seriously, I wouldn't want to burden you with this is everyone else if using the validation set for final numbers. I will leave it up to you whether you think having a test partition would improve the paper.  The block_size_factor=1 results are interesting. Maybe you can find space to include them in the final paper. Also, you write "we first estimate the block_size_factor to use (typically 0.5)". I'd like to see more details on how this estimation happened. Also, I can't find drop_block_size for figure 4 (it's probably 0.5 ...).   In figure 4, right side, "after sum" is missing an 'm'.