We describe a simple and general neural network weight compression approach, in which the network parameters (weights and biases) are represented in a "latent" space, amounting to a reparameterization. This space is equipped with a learned probability model, which is used to impose an entropy penalty on the parameter representation during training, and to compress the representation using a simple arithmetic coder after training. Classification accuracy and model compressibility is maximized jointly, with the bitrate-accuracy trade-off specified by a hyperparameter. We evaluate the method on the MNIST, CIFAR-10 and ImageNet classification benchmarks using six distinct model architectures. Our results show that state-of-the-art model compression can be achieved in a scalable and general way without requiring complex procedures such as multi-stage training.
The authors propose a novel, information theoretic approach to learning compressed neural networks, whereby model parameters $\theta$ are expressed in an (approximately discrete) latent space as $\theta = f_{\psi}(\phi)$. During training, "reparameterizations" $\Phi$ are treated as draws from a generative distribution $q(\phi)$. As per the source coding theorem, penalizing the entropy of $q$ (more accurately, of $\hat{q}$) directly promotes compressible choices of $\Phi$ and, therefore, of $\Theta$. After training, arithmetic coding of $q$ is used to compress $\Phi$. Empirical results on a suite of common benchmarks demonstrates the proposed algorithm gracefully trades off between model performance and size.
Frequency Domain Interpretation:
  Here, I am operating off of the assumptions that, in Sect 3.2 paragraph 3, $\phi_{w}$ should be $\psi_{w}$. If so, can "Our Method (DFT)" also be interpreted as frequency domain SQ (of convolutional filters)? When trained on natural images, CNN filters are typically "smooth" (see any visualization thereof) and this smoothness translates as a prior of sorts on $\Phi_{w}$. This insight was previously explored in [1, 2] for purposes of compressing CNNs. Does evidence of this manifest in your experiments? For example, do the empirical distributions of high- and low-frequency components of $\phi_{w}$ differ?
Feedback:
  The authors propose a general framework where in (seemingly) arbitrary decoders $f$ can be used for purposes of "reparameterization". It is therefore somewhat disappointing that, in practice, $f$ is restricted to affine transformations and, in some cases, even fixed. Memory issues notwithstanding, it is unclear how well complex decoders $f$ could be jointly learned with (surrogate) encodings $\tilde{\Phi}$ and density models $\tilde{q}$.
  The lack of comparison between the proposed method's test-time throughput and that of its baselines leaves open questions regarding potential real-world use cases. Much of the time, methods for compressing neural networks divide their attention between both memory footprint and prediction speed. Outright ignoring this aspect of the problem seems odd. As a reviewer, I would much rather see the authors take a proactive approach in addressing this issue.
Questions:
  - What do results look like if convolutional decoders $f_{\text{conv}}$ are jointly learned during training?
  - How sensitive is the algorithm to different groups assignments? Can assignments be reliably made by simply looking at the architecture?
Minor Comments:
  - The general notation of the paper is cumbersome at times, can $\phi$ to be changed to, e.g., $\tilde{\theta}$? Similarly, can notation such as $\theta_{k, W}$ be simplified as $W^{k}$?
  - Section 2 jumps around a bit. Consider trying to order things "chronologically" such that training comes before compression? By the same token, details regarding "Model Partitioning" could be move to the end of the section such that the preceding material just refers to $\theta$ more abstractly?
  - Some additional details regarding training (esp. discretization and model distributions $\tilde{q}$) would be appreciated.
[1] "Compressing convolutional neural networks in the frequency domain", Chen et al. 2016
[2] "CNNpack: Packing convolutional neural networks in the frequency domain", Wang et al. 2016