The Reversible Residual Network: Backpropagation Without Storing Activations
Deep residual networks (ResNets) have significantly pushed forward the state-ofthe-art on image classification, increasing in performance as networks grow both deeper and wider. However, memory consumption becomes a bottleneck, as one needs to store the activations in order to calculate gradients using backpropagation. We present the Reversible Residual Network (RevNet), a variant of ResNets where each layer's activations can be reconstructed exactly from the next layer's. Therefore, the activations for most layers need not be stored in memory during backpropagation. We demonstrate the effectiveness of RevNets on CIFAR-10, CIFAR-100, and ImageNet, establishing nearly identical classification accuracy to equally-sized ResNets, even though the activation storage requirements are independent of depth.
The paper introduces a reversible block for residual networks that has the benefit of not needing to store all of the forward pass activations for the backward pass. This enables training of larger models or minibatches, as the memory footprint during training is smaller.
As the memory of GPUs is often a bottleneck when training very large models, the paper is a welcome addition to the line of literature on decreasing this footprint. By training on both CIFAR and ImageNet, the experiments focus on two widely used benchmarks. The experiments seem solid and achieve compelling results. The authors also confirm empirically that numerical instabilities are not a significant problem, which is good.
To make the paper even stronger, I would have been interested in seeing even more experiments. It is good that the model is tested both on CIFAR (10 and 100) and ImageNet, but for example SVHN would have been a fairly "cheap" addition to the set of standard benchmarks. The performance on RNNs, that the authors also discuss, would be very interesting to see.
In addition, the main concern I have with using the model in practice is the potential drawback of not discarding information (hence the reversibility) - I'm not sure exactly what form that discussion would take, but a slightly deeper discussion on that issue would be helpful. The second paper on NICE might give some ideas for how to do that.
A minor issue is also that it is somewhat unclear to the reader how large a typical memory footprint of the activations is compared to the footprint of the weights for a particular minibatch size in the resnets. This obviously depends on many factors, but it would be really helpful (for instance in the appendix) to make some back-of-the-envelope calculations that would give an indication of e.g. how much deeper models or how much larger minibatches one could train, given an optimized code, for typical model sizes.
The paper is generally well written, although some minor typos are still present (p1: "simply by widening (increasing the convolutions' filter count) fewer layer" should probably read "...layers", p2: "...derivative defines the the effect..." has one "the" too much, p2: "change to v_i, taking into account the indirect effects through the descendants of v_k" should probably read "... descendants of v_i") which will be easy to fix by carefully rereading the paper a few times.
The drawbacks are however fairly minor, and the strength of the paper is that the idea is fairly simple, has been shown with sufficient empirical results to work, and is well presented. In my view, this is an idea that would interest the NIPS audience.