RELAXED QUANTIZATION FOR DISCRETIZED NEURAL NETWORKS
Neural network quantization has become an important research area due to its great impact on deployment of large models on resource constrained devices. In order to train networks that can be effectively discretized without loss of performance, we introduce a differentiable quantization procedure. Differentiability can be achieved by transforming continuous distributions over the weights and activations of the network to categorical distributions over the quantization grid. These are subsequently relaxed to continuous surrogates that can allow for efficient gradient-based optimization. We further show that stochastic rounding can be seen as a special case of the proposed approach and that under this formulation the quantization grid itself can also be optimized with gradient descent. We experimentally validate the performance of our method on MNIST, CIFAR 10 and Imagenet classification.
Summary
=======
This paper introduces a method for learning neural networks with quantized weights and activations. The main idea is to stochastically – rather than deterministically – quantize values, and to replace the resulting categorical distribution over quantized values with a continuous relaxation (the "concrete distribution" or "Gumbel-Softax distribution"; Maddison et al., 2016; Jang et al., 2016). Good empirical performance is demonstrated for LeNet-5 applied to MNIST, VGG applied to CIFAR-10, and MobileNet and ResNet-18 applied to ImageNet.
Review
======
Relevance:
Training non-differentiable neural networks is a challenging and important problem for several applications and a frequent topic at ICLR.
Novelty:
Conceptually, the proposed approach seems like a straight-forward application/extension of existing methods, but I'm unaware of any paper which uses the concrete distribution for the express purpose of improved efficiency as in this paper. There is a thorough discussion of related work, although I was missing Williams (1992), who used stochastic rounding before Gupta et al. (2015), and Soudry et al. (2014), who introduced a Bayesian approach to deal with discrete weights and activations.
Results:
The empirical work is thorough, achieving state-of-the-art results in several classification benchmarks. It would be interesting to see how well these methods perform in other tasks (e.g., compression or even regression), even though the literature on quantization seems to focus on classification.
Clarity:
The paper is well written and clear.