Published as a conference paper at ICLR 2018 VARIATIONAL IMAGE COMPRESSION WITH A SCALE HYPERPRIOR
We describe an end-to-end trainable model for image compression based on variational autoencoders. The model incorporates a hyperprior to effectively capture spatial dependencies in the latent representation. This hyperprior relates to side information, a concept universal to virtually all modern image codecs, but largely unexplored in image compression using artificial neural networks (ANNs). Unlike existing autoencoder compression methods, our model trains a complex prior jointly with the underlying autoencoder. We demonstrate that this model leads to state-of-the-art image compression when measuring visual quality using the popular MS-SSIM index, and yields rate-distortion performance surpassing published ANN-based methods when evaluated using a more traditional metric based on squared error (PSNR). Furthermore, we provide a qualitative comparison of models trained for different distortion metrics.
Authors propose a transform coding solution by extending the work in Balle 2016. They define an hyperprior for the entropy coder to model the spatial relation between the transformed coefficients.    
The paper is well written, although I had trouble following some parts. The results of the proposal are state-of-the-art, and there is an extremely exhaustive comparison with many methods.
In my opinion the work has a good quality to be presented at the ICLR. However, I think it could be excellent if some parts are improved. Below I detail some parts that I think could be improved.
*** MAIN ISSUES
I have two main concerns about motivation that are related. The first refers to hyperprior motivation. It is not clear why, if GDN was proposed to eliminate statistical dependencies between pixels in the image, the main motivation is that GDN coefficients are not independent. Perhaps this confusion could be resolved by broadening the explanation in Figure 2. My second concern is that it is not clear why it is better to modify the probability distribution for the entropy encoder than to improve the GDN model. I think this is a very interesting issue, although it may be outside the scope of this work. As far as I know, there is no theoretical solution to find the right balance between the complexity of transformation and the entropy encoder. However, it would be interesting to discuss this as it is the main novelty of the work compared to other methods of image compression based on deep learning. 
*** OTHER ISSUES 
INTRODUCTION
-"...because our models are optimized end-to-end, they minimize the total expected code length by learning to balance the amount of side information with the expected improvement of the entropy model." 
I think this point is very interesting, it would be good to see some numbers of how this happens for the results presented, and also during the training procedure. For example, a simple comparison of the number of bits in the signal and side information depending on the compression rate or the number of iterations during model training.  
COMPRESSION WITH VARIATIONAL MODELS
- There is something missing in the sentence: "...such as arithmetic coding () and transmitted..."
- Fig1. To me it is not clear how to read the left hand schemes. Could it be possible to include the distributions specifically? Also it is strange that there is a \tiled{y} in both schemes but with different conditional dependencies. Another thing is that the symbol ψ appears in this figure and is not used in section 2. 
- It would be easier to follow if change the symbols of the functions parameters by something like \theta_a and \theta_s.
- "Distortion is the expected difference between..." Why is the "expected" word used here? 
- "...and substituting additive uniform noise..." is this phrase correct? Are authors is Balle 2016 substituting additive uniform noise?   
- In equation (1), is the first term zero or constant? when talking about equation (7) authors say "Again, the first term is constant,...".
- The sentence "Most previous work assumes..." sounds strange.
- The example in Fig. 2 is extremely important to understand the motivation behind the hyperprior but I think it needs a little more explanation. This example is so important that it may need to be explained at the beginning of the work. Is this a real example, of a model trained without and with normalization? If so specify it please. Why is GDN not able to eliminate these spatial dependencies? Would these dependencies be eliminated if normalization were applied between spatial coefficients? Could you remove dependencies with more layers or different parameters in the GDN?
INTRODUCTION OF A SCALE HYPERPRIOR
- TYPO "...from the center pane of..."
- "...and propose the following extension of the model (figure 3):" there is nothing after the colon. Maybe there is something missing, or maybe it should be a dot instead of a colon. However to me there is a lack of explanation about the model. 
 
RESULTS
- "...,the probability mass functions P_ŷi need to be constructed “on the fly”..."
How computationally costly is this? 
- "...batch normalization or learning rate decay were found to have no beneficial effect (this may be due to the local normalization properties of GDN, which contain global normalization as a special case)."
This is extremely interesting. I see the connection for batch normalization, but not for decay of the learning rate. Please, clarify it. Does this mean that when using GDN instead of regular nonlinearity we no longer need to use batch normalization? Or in other words, do you think that batch normalization is useful only because it is special case of GSN? It would be useful for the community to assess what are the benefits of local normalization versus global normalization.
- "...each of these combinations with 8 different values of λ in order to cover a range of rate–distortion tradeoffs." 
Would it be possible with your methods including \lambda as an input and the model parameters as side information?
- I guess you included the side information when computing the total entropy (or number of bits), was there a different way of compressing the image and the side information?
- Using the same metrics to train and to evaluate is a little bit misleading. Evaluation plots using a different perceptual metric would be helpful. 
-"Since MS-SSIM yields values between 0 (worst) and 1 (best), and most of the compared methods achieve values well above 0.9, we converted the quantity to decibels in order to improve legibility." 
Are differences of MS-SSIM with this conversion significant? Is this transformation necessary, I lose the intuition. Besides, probably is my fault but I have not being able to "unconvert" the dB to MS-SSIM units, for instance 20*log10(1)= 20 but most curves surpass this value.  
- "..., results differ substantially depending on which distortion metric is used in the loss function during training."   
It would be informative to understand how the parameters change depending on the metric employed, or at least get an intuition about which set of parameters adapt more g_a, g_s, h_a and h_s.
- Figs 5, 8 and 9. How are the curves aggregated for different images? Is it the mean for each rate value? Note that depending on how this is done it could be totally misleading.
 	
- It would be nice to include results from other methods (like the BPG and Rippel 2017) to compare with visually.
RELATED WORK
Balle et al. already published a work including a perceptual metric in the end-to-end training procedure, which I think is one of the main contributions of this work. Please include it in related work:
"End-to-end optimization of nonlinear transform codes for perceptual quality." J. Ballé, V. Laparra, and E.P. Simoncelli. PCS: Picture Coding Symposium, (2016) 
DISCUSSION
First paragraphs of discussion section look more like a second section of "related work". 
I think it is more interesting if the authors discuss the relevance of putting effort into modelling hyperprior or the distribution of images (or transformation). Are these things equivalent? Or is there some reason why we can't include hyperprior modeling in the g_a transformation? For me it is not clear why we should model the distribution of outputs as, in principle, the g_a transformation has to enforce (using the training procedure) that the transformed data follow the imposed distribution. Is it because the GDN is not powerful enough to make the outputs independent? or is it because it is beneficial in compression to divide the problem into two parts?   
REFERENCES
- Balle 2016 and Theis 2017 seem to be published in the same conference the same year. Using different years for the references is confusing.
- There is something strange with these references
Ballé, J, V Laparra, and E P Simoncelli (2016). “Density Modeling of Images using a Generalized
Normalization Transformation”. In: Int’l. Conf. on Learning Representations (ICLR2016). URL :
https://arxiv.org/abs/1511.06281.
Ballé, Valero Laparra, and Eero P. Simoncelli (2015). “Density Modeling of Images Using a Gen-
eralized Normalization Transformation”. In: arXiv e-prints. Published as a conference paper at
the 4th International Conference for Learning Representations, San Juan, 2016. arXiv: 1511.
06281.
– (2016). “End-to-end Optimized Image Compression”. In: arXiv e-prints. 5th Int. Conf. for Learn-
ing Representations.