Swapout: Learning an ensemble of deep architectures
We describe Swapout, a new stochastic training method, that outperforms ResNets of identical network structure yielding impressive results on CIFAR-10 and CIFAR-100. Swapout samples from a rich set of architectures including dropout [20], stochastic depth [7] and residual architectures [5,6] as special cases. When viewed as a regularization method swapout not only inhibits co-adaptation of units in a layer, similar to dropout, but also across network layers. We conjecture that swapout achieves strong regularization by implicitly tying the parameters across layers. When viewed as an ensemble training method, it samples a much richer set of architectures than existing methods such as dropout or stochastic depth. We propose a parameterization that reveals connections to exiting architectures and suggests a much richer set of architectures to be explored. We show that our formulation suggests an efficient training method and validate our conclusions on CIFAR-10 and CIFAR-100 matching state of the art accuracy. Remarkably, our 32 layer wider model performs similar to a 1001 layer ResNet model.
the layers. The theoretical evidence that a depth of 1000 is required for practical problems is thin. Bengio and Dellaleau argue that circuit efficiency constraints suggest increasing depth is important, because there are functions that require exponentially large shallow networks to compute [1]. Less experimental interest has been displayed in the width of the networks (the number of filters in a convolutional layer). We show that increasing the width of our swapout networks leads to significant improvements in their accuracy; an appropriately wide swapout network is competitive with a deep residual network that is 1.5 orders of magnitude deeper and has more parameters.
Contributions: Swapout is a novel stochastic training scheme that can sample from a rich set of architectures including dropout, stochastic depth and residual architectures as special cases. Swapout improves the performance of the residual networks for a model of the same depth. Wider but much shallower swapout networks are competitive with very deep residual networks.
This paper proposes a new form of the popular dropout technique for regularization of deep neural networks.  In particular, this approach develops a dropout strategy for a combination of recent developments in deep learning (residual networks, stochastic depth networks).  The algorithm stochastically chooses between dropping out residual connections, feedforward connections and both.  The authors demonstrate that this strategy empirically improves on these aforementioned techniques on a convolutional networks applied to CIFAR-10 and CIFAR-100.
This is a generally well written paper that develops a neat novel idea for regularization of deep networks.  I think the presented idea could be impactful in that it may prove to be very useful in practice to many researchers and practitioners.  The presented empirical evaluation demonstrates that the approach is quite promising.  The idea is, however, not tremendously novel in light of recent work (res-nets and stochastic depth networks) and it would have been nice to see comparison on some more interesting benchmarks.  small typo in the abstract: exiting -> existing  I appreciate the additional analysis of the test time statistics of the approach.  Why can't you estimate the test time statistics empirically on a validation set?    I really appreciate the tidbits on why dropout and swapout interact poorly with batch normalization.     It's useful to know that you don't have to average over very many sampled dropouts (swapouts).  I think this is a neat additional analysis and rather useful to the community.  Why do the authors first do exactly 196 and then 224 epochs before decaying the learning rate?  Normally such specific choices would arouse suspicion except in this case I expect it doesn't make much difference (e.g. between 196 and a round number like 200).  Some additional detail on how the hyperparameters were chosen seems necessary here.  I think it would have significantly improved the paper if experiments were run on more interesting problems.  CIFAR-10 seems to be exhausted at this point.  The current numbers are beyond the human level error of ~6%, which is rather worrying.  Papers that beat this number are either overfitting to the test set or to the original labeler.  Either way, it doesn't seem like a particularly useful benchmark anymore.  One of the major reasons why res-nets were so well received was because they achieved state-of-the-art on the much larger and more challenging imagenet.  A major result of the paper is that we might not need as deep networks as originally thought.  However, it doesn't seem surprising (to me at least) that one wouldn't need an especially deep network for the 32x32 images in the CIFAR dataset (whereas this may be a different story for much larger images with more classes).  As far as I can tell, the authors do not actually compare to the state-of-the-art on CIFAR-10.  The All-convolutional-net paper achieved a lower error.  Could the authors explain why these baselines are not included?  The paper states that it compares to 'fair' baselines but this seems like a somewhat qualified result.  e.g. As far as I can tell, the standard data-augmentation tricks are applied in this work as well.  There is a lot of extra space available in this paper (see e.g. section 3.2).  This leaves lots of space for an additional table of experimental results on e.g. imagenet.