Quantizing large Neural Networks (NN) while maintaining the performance is highly desirable for resource-limited devices due to reduced memory and time complexity. NN quantization is usually formulated as a constrained optimization problem and optimized via a modified version of gradient descent. In this work, by interpreting the continuous parameters (unconstrained) as the dual of the quantized ones, we introduce a Mirror Descent (MD) framework (Bubeck (2015)) for NN quantization. Specifically, we provide conditions on the projections (i.e., mapping from continuous to quantized ones) which would enable us to derive valid mirror maps and in turn the respective MD updates. Furthermore, we discuss a numerically stable implementation of MD by storing an additional set of auxiliary dual variables (unconstrained). This update is strikingly analogous to the popular Straight Through Estimator (STE) based method which is typically viewed as a "trick" to avoid vanishing gradients issue but here we show that it is an implementation method for MD for certain projections. Our experiments on standard classification datasets (CIFAR-10/100, TinyImageNet) with convolutional and residual architectures show that our MD variants obtain fully-quantized networks with accuracies very close to the floating-point networks.
The paper proposes to use the mirror descent algorithm for the binary network. The key point is Theorem 3.1, which enables the mirror map. The paper is easy to read and follow, and the main contributions are clearly stated.
However, I suggest a weak rejection of this paper. The reasons are
Q1. As Review #3, it is better for authors to provide more theoretical analysis, which better includes the nonconvex objective function and the effect of annealing. 
Q2. It is not clear to me, why mirror descent is better than proximal gradient descent, i.e., proxQuant, in this application. The authors repeatedly claim "MD allows gradient descent to be performed on a more general non-Euclidean space". This cannot be told by Table 1, which is just overall performance. So, it is better to empirically show this point by an ablation study.
Q3. Since the technical contributions are not enough, I expect more experimental comparisons.
- Could the authors perform experiments on ImageNet?
- While VGG and ResNet are taken as a protocol for experimental comparison, it is better to do an extra comparison with STOA networks. VGG and ResNet are too old and easy to be compressed, compression these networks are of little practical values. EfficientNet [1], Mobilenets [2], and Shufflenet [3] can be good ones. The paper will be more convincing with these methods.
[1]. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
[2]. Mobilenets: Efficient convolutional neural networks for mobile vision applications
[3]. Shufflenet: An extremely efficient convolutional neural network for mobile devices