Deep neural networks with discrete latent variables offer the promise of better symbolic reasoning, and learning abstractions that are more useful to new tasks. There has been a surge in interest in discrete latent variable models, however, despite several recent improvements, the training of discrete latent variable models has remained challenging and their performance has mostly failed to match their continuous counterparts. Recent work on vector quantized autoencoders (VQ-VAE) has made substantial progress in this direction, with its perplexity almost matching that of a VAE on datasets such as CIFAR-10. In this work, we investigate an alternate training technique for VQ-VAE, inspired by its connection to the Expectation Maximization (EM) algorithm. Training the discrete autoencoder with EM and combining it with sequence level knowledge distillation alows us to develop a nonautoregressive machine translation model whose accuracy almost matches a strong greedy autoregressive baseline Transformer, while being 3.3 times faster at inference.
Summary: 
This paper presents a new training algorithm for vector-quantized autoencoders (VQVAE), a discrete latent variable model akin to continuous variational autoencoders.
The authors propose a soft-EM training algorithm for this model, that replaces hard assignment of latent codes to datapoints with a weighted soft-assignment.
Overall the technical writing in the paper is sloppy, and the presentation of the generative model takes the form of an algorithmic description of the training algorithm, rather than being a clear definition of the generative model itself.
The technical presentation of the work by the authors starts only at page 5 (taking less than a full page), after several pages of imprecise presentation of previous and related work. The paper could be significantly improved by making this preceding material more concise and rigorous. 
Quantitative experimental evaluation is limited to a machine translation task, which is rather uncommon in the literature on generative latent variable models. I would expect evaluation in terms of held-out data log-likelihood (ie bits-per-dimension) used in probabilistic generative models, and possibly also using measures from the GAN literature such as inception scores. Datasets that are common include CIFAR-10 and resized variants of the imagenet dataset. 	 
Specific comments:
- Please adhere to the ICLR template bibliography style, which is far more readable than the style that you used. 
- Figure 1 does not seem to be referenced in the text. 
- The last paragraph of section 2.1 is unclear. It mentions a sampling a sequence of latent codes. The notion of sequentiality has not been mentioned before, and it is not clear what it refers to in the context of the model defined so far up to that point. 
- The technical notation is very sloppy. 
* In numerous places the paper refers to the joint distribution P(x1,…,x_n, z1, …, zn) without defining that the distribution factorizes across the samples (xi,zi), and without specifying the forms of p(zi) and p(xi|zi). 
* This makes that claims such as “computing the expectation in the M step (Equation 11) is computationally infeasible” are not verifiable. 
- Please be clear about how much is gained by replacing the exact M-step with a the one based on the samples from the posterior computed in the E-step. 
- What is the reason to decode the weighted average of the embedding vectors, rather than decoding all of them, and updating the decoder in a weighted manner?
- reference 14 for Variational autoencoders is incorrect, please use the following citation instead: 
@InProceedings{kingma14iclr,
  Title                    = {Auto-Encoding Variational {B}ayes},
  Author                   = {D. Kingma and M. Welling},
  Booktitle                = {{ICLR}},
  Year                     = {2014}
}
- The related work section (4) provides a rather limited overview of relevant related work. 
Half of it is dedicated to recent advances in machine translation, which does not bear a direct connection to the technical material presented in section 3.
- There is no justification of using *causal* self-attention on the source embedding, is this a typo?
- As for the experimental evaluation results: it seems that distillation is a much more critical factor to achieve good performance than the proposed EM training of the VQ-VAE model. Unfortunately, this fact goes unmentioned when discussing the experimental results. 
- What is the significance of the observed differences in BLEU scores? Please report average performance and standard deviations over several runs with randomized parameter initialization and batch scheduling. 
- It seems that the tuning of the number of discrete latent codes (table 2 in appendix) and other hyper-parameters (table 3 in appendix) was done on the test set, which is also used to compare to related work. A separate validation set should be used for hyper parameter tuning in machine learning experiments.
- It seems that all curves in figure 3 collapse from about 45 BLEU to values around 17 BLEU, why is this? The figure is hard to read since poor quality, and curves that are superposed.