INTEGER NETWORKS FOR DATA COMPRESSION WITH LATENT-VARIABLE MODELS
We consider the problem of using variational latent-variable models for data compression. For such models to produce a compressed binary sequence, which is the universal data representation in a digital world, the latent representation needs to be subjected to entropy coding. Range coding as an entropy coding technique is optimal, but it can fail catastrophically if the computation of the prior differs even slightly between the sending and the receiving side. Unfortunately, this is a common scenario when floating point math is used and the sender and receiver operate on different hardware or software platforms, as numerical round-off is often platform dependent. We propose using integer networks as a universal solution to this problem, and demonstrate that they enable reliable cross-platform encoding and decoding of images using variational models.
The paper presents a very important problem of utilizing a model on different platforms with own numerical round-offs. As a result, a model run on a different hardware or software than the one on which it was trained could completely fail due to numerical rounding-off issues. This problem has been considered in various papers, however, the classification task was mainly discussed. In this paper, on the other hand, the authors present how the numerical rounding-off issue could be solved in Latent-Variable Models (LVM).
In order to cope with the numerical rouding-off issue, the authors propose to use integer networks. They consider either quantized ReLU (QReLU) or quantized Tanh (Qtanh). Further, in order to properly train the integer NN, they utilize a bunch of techniques proposed in the past, mainly (Balle, 2018) and (Balle et al., 2018). However, as pointed out in the paper, some methods prevent training instabilities (e.g., Eqs. 18 and 19). All together, the paper tackles very important problem and proposes very interesting solution by bringing different techniques proposed for quantized NNs together .
Pros:
+ The paper is well-written.
+ The considered problem is of great importance and it is rather neglected in the literature.
+ The experiments are properly carried out.
+ The obtained results are impressive.
Cons:
- A natural question is whether the problem could be prevented by post-factum quantization of a neural network. As pointed out in the Discussion section, such procedure failed. However, it would be beneficiary to see an empirical evidence for that.
- It would be also interesting to see how a training process of an integer NN looks like. Since the NN is quantized, instabilities during training might occur. Additionally, its training process may take longer (more epochs) than a training of a standard (float) NN. An exemplary plot presenting a comparison between an integer NN training process and a standard NN training process would be highly appreciated.
- (Minor remark). The paper is well-written, however, it would be helpful to set the final learning algorithm. This would drastically help in reproducibility of the paper.
--REVISION--
After reading the authors response and looking at the new version of the paper I decided to increase my score. The paper tackles very important problem and I strongly believe it should be presented during the conference.