This paper proposes a powerful regularization method named ShakeDrop regularization. ShakeDrop is inspired by Shake-Shake regularization that decreases error rates by disturbing learning. While Shake-Shake can be applied to only ResNeXt which has multiple branches, ShakeDrop can be applied to not only ResNeXt but also ResNet, Wide ResNet and PyramidNet in a memory efficient way. Important and interesting feature of ShakeDrop is that it strongly disturbs learning by multiplying even a negative factor to the output of a convolutional layer in the forward training pass. The effectiveness of ShakeDrop is confirmed by experiments on CIFAR-10/100 and Tiny ImageNet datasets.  (Huang et al., 2016)  and Shake-Shake (Gastaldi, 2017) are known to be effective regularization methods for ResNet and its improvements. Among them, Shake-Shake applied to ResNeXt is the one achieving the lowest error rates on CIFAR-10/100 datasets (Gastaldi, 2017).
Shake-Shake, however, has following two drawbacks. (1) Shake-Shake can be applied to only multibranch architectures (i.e., ResNeXt). (2) Shake-Shake is not memory efficient. Both drawbacks come from the same root. That is, Shake-Shake requires two branches of residual blocks to apply. If it is true, it is not difficult to conceive its solution: a similar disturbance to Shake-Shake on a single residual block. It is, however, not trivial to realize it.
The current paper addresses the problem of realizing a similar disturbance to Shake-Shake on a single residual block, and proposes a powerful regularization method, named ShakeDrop regularization. While the proposed ShakeDrop is inspired by Shake-Shake, the mechanism of disturbing learning is completely different. ShakeDrop disturbs learning more strongly by multiplying even a negative factor to the output of a convolutional layer in the forward training pass. In addition, a different factor from the forward pass is multiplied in the backward training pass. As a byproduct, however, learning process gets unstable. Our solution to this problem is to stabilize the learning process by employing ResDrop in a different usage from the usual. Based on experiments using various base network architectures, we reveal the condition that the proposed ShakeDrop successfully works.
The paper proposes ShakeDrop regularization, which is essentially a combination of the PyramidDrop and Shake-Shake regularization. The procedure consists of essentially weighting the residual branch with a random weight, in the style of Shake-Shake, where the weight is sampled from a mixture of uniform distribution in [-1, 1] and delta at 1, such that the mixture of those two distributions varies linearly with layer depth, in the style of PyramidDrop. In the style of Shake-Shake, a different random weight (in [0, 1]) is used for the backward pass. The most surprising part is that that the forward weight can be negative thus inverting the output of a convolution. Apparently the goal is to "disturb" the training, and the procedure yields state-of-the-art results on CIFAR-10/100.
Positives:
- Results: state-of-the-art on CIFAR-10/100
Negatives:
1. No real motivation on why should this work. I guess the motivation is the mixture of PyramidDrop and Shake-Shake motivations, but the main surprising part (forward weight can be negative) is not motivated at all. There is a tiny bit of discussion at the very end, section 4.4, where authors examine the training loss (showing it's non-zero so less overfitting) and mean/variance of gradients (increased). However, this doesn't really satisfy me - it is clear that more disturbance will cause this behaviour, but that doesn't mean any disturbance is good, e.g. if I always apply the negative weight and make my model weights go in the wrong direction, I'm pretty sure training loss and gradients will be even larger, but it's a bad idea to do.
2. I'm concerned with the "weird trick that happens to work on CIFAR" line of work (not saying that this paper is the only offender) - are these methods actually useful and generalizable to other problems, or are we overfitting on CIFAR and creating MNIST v2.0 ? It would be nice to demonstrate that this regularization works in at least one more problem, maybe ImageNet, though maybe regularization is not needed there but just find one more dataset that needs regularization and test this on that.
3. The paper doesn't explain well what is the problem with Shake-Shake and memory. I see that the author of Shake-Shake has made a comment on this and that makes a lot of sense, i.e. there is no memory issue, just because there are 2x branches doesn't mean shake-shake needs 2x memory as it can use less capacity=memory to achieve the same performance. So it seems the main premise of the paper - "let's apply Shake-Shake to deeper models but we need to come up with a modified method because Shake-Shake cannot be applied due to memory problems" - seems wrong.
4. The writing quality is quite bad, it is very hard to understand what authors mean in parts of the text. E.g. at two places "it has almost the same residual block as Eqn. (1)" - how is it "almost"? Below equation 5, it is never specified that alpha and beta are sampled uniformly(?) from those ranges, one could think that alpha and beta are fixed constants that take a specific value that is in that range. There are also various grammatical errors such as "is expected to be powerful but slight memory overhead" or "which is introduced essence", etc.
Smaller comments:
- Isn't it surprising that alpha in [-1, 1] and beta in [0, 1] works well, but alpha in [0, 1] and beta in [-1, 1] works much worse? The two important cases, (alpha negative, beta positive) and (alpha positive, beta negative), seem to me like they are conceptually very similar.
- End of section 4.1, should it be b_l as p_L is a constant and b_l is what is sampled?
- I don't like that exactly the same text is repeated 3 times (abstract, end of intro, end of 1.1) and in very short distance from each other - repeating the same words 3 times doesn't make the reader understand it better, slight rephrasing is much more beneficial.
Overall:
Good to know that this method sets the new state of the art on CIFAR-10/100, so as such it should be of interest to the community to be available online (arXiv). But with fairly little novelty (is a combination of 2 methods), very little insights of why this should work at all (especially the negative scaling coefficient which is the only extra thing that one learns from this paper, since the rest is a combination of PyramidDrop and Shake-Shake), no idea on whether the method would work outside of the CIFAR-world, and bad quality of the text - I don't think the manuscript is sufficiently good for ICLR.