PRACTICAL LOSSLESS COMPRESSION WITH LATENT VARIABLES USING BITS BACK CODING
Deep latent variable models have seen recent success in many data domains. Lossless compression is an application of these models which, despite having the potential to be highly useful, has yet to be implemented in a practical manner. We present 'Bits Back with ANS' (BB-ANS), a scheme to perform lossless compression with latent variable models at a near optimal rate. We demonstrate this scheme by using it to compress the MNIST dataset with a variational autoencoder model (VAE), achieving compression rates superior to standard methods with only a simple VAE. Given that the scheme is highly amenable to parallelization, we conclude that with a sufficiently high quality generative model this scheme could be used to achieve substantial improvements in compression rate with acceptable running time. We make our implementation available open source at https://github.com/bits-back/bits-back.
This paper is built on a simple but profound observation: Frey's bits-back coding algorithm can be implemented much more elegantly when replacing arithmetic coding (AC) with asymmetric numerical systems (ANS), a much more recent development not known at the time, simply due to the fact that it encodes symbols in a stack-like fashion rather than queue-like.
This simple observation makes for an elegantly written paper, with promising results on MNIST. I truly enjoyed reading it, and I'm convinced that it will spark some very interesting further work in the field of compression with latent-variable models.
Having said that, I would like to point out some possible limitations of the proposed approach, which I hope the authors will be able to address/clarify:
1. At the beginning of section 2.1, the authors define the symbols as chained conditionals prod_n p(s_n | s_1 ... s_n-1), which is generally permissible in AC as well as ANS, as long as the decoding order is taken into account. That is, in AC, the symbols need to be encoded starting with the first symbol in the chain (s_1), while in ANS, the symbols must be encoded starting with the last symbol in the chain, because the decoding order is inverted.
In their description of BB-ANS, the authors omit the discussion of conditional chains. It is unclear to me if a conditioning of the symbols is feasible in BB-ANS due to the necessity to maintain a strict decoding order. It would be very helpful if the authors could clarify this, and update the paper accordingly, because this could present a serious limitation. For instance, the authors simply extrapolate the performance of their method to PixelVAE; however, this model is autoregressive, so a conditioning of symbols seems necessary. Similarly, in appendix A, the authors mention the work of Minnen et al. (2018), where the same situation would apply, albeit one probabilistic level higher (on encoding/decoding the latents with an autoregressive prior).
2. Furthermore, in both cases (PixelVAE and Minnen et al.), the symbols (s) and latents (y) are defined as jointly conditioned on each other (i.e., computing the posterior on one element of y requires knowledge of all elements of s, and computing the likelihood on one element of s requires knowledge of all elements of y). This seems to imply that all operations pertaining to one data vector (i.e. to one image) would have to be done in a monolithic fashion, i.e.: first sample all elements of y from the stack, then encode all elements of s, and then encode all elements of y. Hence, if the goal is to compress only one image, the algorithm would never get to the point of reusing the "bits back", and the overhead of BB-ANS would be prohibitive. It seems that in the MNIST experiments, the authors avoid this problem by always encoding a large number of images at a time, such that the overhead is amortized.
3. Similarly, although the compression of continuous-valued variables up to arbitrary precision is an exciting development and I do not wish to undermine the importance of this finding, it should be noted that the finer the quantization gets, the larger the potential overhead of the coding scheme will grow. In practice, this would make it necessary to encode more and more images together, in order to still benefit from the method. This would be a good point to make in the discussion.
4. The authors state in the appendix that learned compression methods like Ballé et al. (2018) and Minnen et al. (2018) could be improved by using BB-ANS. The potential gain of BB-ANS for these models seems rather small, though, as the entropy of y must be larger or equal to the entropy of y conditioned on s: H[y] >= H[y|s], the latter of which should represent the potential coding gain. Ballé et al. (2018), however, found that the bits used to encode the hierarchical prior (i.e. H[y]) is only a small fraction of the total bitrate, thus upper bounding the potential gains for this type of model.
Overall, I think this is a well-written, important and elegant paper, and I would like to see it accepted at this conference. If the authors can satisfactorily address some of the above potential limitations, it might turn out to be even better.