Training Neural Networks Using Features Replay
Training a neural network using backpropagation algorithm requires passing error gradients sequentially through the network. The backward locking prevents us from updating network layers in parallel and fully leveraging the computing resources. Recently, there are several works trying to decouple and parallelize the backpropagation algorithm. However, all of them suffer from severe accuracy loss or memory explosion when the neural network is deep. To address these challenging issues, we propose a novel parallel-objective formulation for the objective function of the neural network. After that, we introduce features replay algorithm and prove that it is guaranteed to converge to critical points for the non-convex problem under certain conditions. Finally, we apply our method to training deep convolutional neural networks, and the experimental results show that the proposed method achieves faster convergence, lower memory consumption, and better generalization error than compared methods.
## Summary  The paper proposes a novel algorithm for backpropagation in neural networks, called ‘Feature Replay’ (FR) which speeds up the gradient computation in the backwards pass at the expense of some accuracy. This is done by splitting the net into K sub-modules in which each sub-layer is updated with an outdated gradient of the (at most) K-shifted residual. The authors claim that FR handles the accuracy vs. speed/memory-tradeoff better than existing algorithms. The authors include a proof of convergence based on several assumptions which the authors then attempt to validate experimentally. The algorithm is tested against two competitors & vanilla backprop on two well worn image classification benchmarks (CIFAR-10/100). Additional insights are provided in the form of memory consumption plots as well as plots that show speed vs. accuracy (num. K).  The paper seems to tackled an important problem in neural network training which is speeding up the training time. While I found the theoretical development of the method convincing, I cannot comment all too much about the choice of competitors and their execution in the experiments. The only proper baseline which is considered is vanilla backprop and DDG, while the second competitor (DNI) oddly seems to never really work.   ## Novelty & originality & Significance I am not an expert in the field so I am unsure about novelty & originality, but I personally have not seen the same algorithm before. Also the general formulation of the problem as a minimization task, which is then solved by an ad-hoc approximation possibly opens up different variations of the proposed algorithm in the future.   ## Clarity The motivation of the paper is clearly written and easy to follow.   ## Major points  - Eq. 8 introduces the constant sigma which, if it exists, bounds the deviation of the approximate gradient to the true one. Figure 3 shows empirical results of sigma. Could the authors please comment on how this is computed? Did they also compute the true gradient at each step? Also, since Eq. 8 assume no mini-batches while in Figure 3 there seems to be one value per epoch only, did the authors average the estimated sigmas? In that case there could be sigmas which are not bounded away from zero, although the mean has a large value.  - The authors emphasize that their algorithm does not suffer from approximation errors as much as competitors if the network is deep. Still, the error gradients seem to be less accurate by construction the deeper the network. How deep can I make the network such that the algorithm is still feasible? Did the authors conduct experiments in that direction? I am asking since the networks considered do not seem to be particularly deep after all.  - l. 109 ff. states that w^t approx w^{t-1} et cetera. Do the authors assume convergence here? If yes, it should be mentioned. Also, in what way is the approximation sign meant? I think there is no unique way of saying that a vector is similar to another one. Could the authors please clarify this paragraph?  - Could the authors comment on how to set K in practice? As a user of the proposed algorithm I would preferably not like to tune it. The paper says ‘we always set K very small’, what does that mean?  - The authors used SGD+momentum in their experiments. Why? Does the method not work with SGD since it introduces more noise to the gradients? SGD+momentum seems to be a sensible choice since it can be seen as averaging out some of the gradient noise, but I’d like to understand if it is essential for the proposed algorithm.  - Did the authors consider to lower the learning rate for DNI a bit? If the gradients are more noisy their elements are also larger in expectation. DNI might have converged with some mild tuning. This might have enabled a fairer comparison.  - Algorithm 1 is helpful.  ## Minor points & typos - l.130 ff. Could you please not use x and y for inputs to f. It is slightly confusing since e.g., y is used for the training targets, too.   ## Post-rebuttal Thank you for the clarifications and the thorough rebuttal. I increased my score. The details: [1] OK.  [2] Thank you for clarifying. If there is a way to include this information in the paper this would be very helpful for interpreting the plot properly. [3] cool. Thank you. [4] Thank you for clarifying. It would be helpful to add a small remark in the text. [5] OK. If there is a way of rephrasing a sentence to say this that would help.  [6] OK. Thank you for including the plot in the rebuttal. That was very helpful.  [7] I see. Thank you [8] Thanks