Published as a conference paper at ICLR 2018 DEEP REWIRING: TRAINING VERY SPARSE DEEP NET- WORKS
Neuromorphic hardware tends to pose limits on the connectivity of deep networks that one can run on them. But also generic hardware and software implementations of deep learning run more efficiently for sparse networks. Several methods exist for pruning connections of a neural network after it was trained without connectivity constraints. We present an algorithm, DEEP R, that enables us to train directly a sparsely connected neural network. DEEP R automatically rewires the network during supervised training so that connections are there where they are most needed for the task, while its total number is all the time strictly bounded. We demonstrate that DEEP R can be used to train very sparse feedforward and recurrent neural networks on standard benchmark tasks with just a minor loss in performance. DEEP R is based on a rigorous theoretical foundation that views rewiring as stochastic sampling of network configurations from a posterior.
This paper presents an iterative approach to sparsify a network already during training. During the training process, the amount of connections in the network is guaranteed to stay under a specific threshold. This is a big advantage when training is performed on hardware with computational limitations, in comparison to "post-hoc" sparsification methods, that compress the network after training.
The method is derived by considering the "rewiring" of an (artificial) neural network as a stochastic process. This perspective is based on a recent model in computational biology but also can be interpreted as a (sequential) monte carlo sampling based stochastic gradient descent approach. References to previous work in this area are missing, e.g.
[1] de Freitas et al., Sequential Monte Carlo Methods to Train Neural Network
Models, Neural Computation 2000
[2] Welling et al., Bayesian Learning via Stochastic Gradient Langevin Dynamics, ICML 2011
Especially the stochastic gradient method in [2] is strongly related to the existing approach.
Positive aspects
- The presented approach is well grounded in the theory of stochastic processes. The authors provide proofs of convergence by showing that the iterative updates converge to a fixpoint of the stochastic process
- By keeping the temperature parameter of the stochastic process high, it can be directly applied to online transfer learning.
- The method is specifically designed for online learning with limited hardware ressources.
Negative aspects
- The presented approach is outperformed for moderate compression levels (by Han's pruning method for >5% connectivity on MNIST, Fig. 3 A, and by l1-shrinkage for >40% connectivity on CIFAR-10 and TIMIT, Fig. 3 B&C). Especially the results on MNIST suggest that this method is most advantageous for very high compression levels. However in these cases the overall classification accuracy has already dropped significantly which could limit the practical applicability.
- A detailled discussion of the relation to previously existing very similar work is missing (see above)
Technical Remarks
Fig. 1, 2 and 3 are referenced on the pages following the page containing the figure. Readibility could be slightly increased by putting the figures on the respective pages.