There are many ways to translate a sentence into another language. Explicit modeling of such uncertainty may enable better model fitting to the data and it may enable users to express a preference for how to translate a piece of content. Latent variable models are a natural way to represent uncertainty. Prior work investigated the use of multivariate continuous and discrete latent variables, but their interpretation and use for generating a diverse set of hypotheses have been elusive. In this work, we drastically simplify the model, using just a single multinomial latent variable. The resulting mixture of experts model can be trained efficiently via hard-EM and can generate a diverse set of hypothesis by parallel greedy decoding. We perform extensive experiments on three WMT benchmark datasets that have multiple human references, and we show that our model provides a better trade-off between quality and diversity of generations compared to all baseline methods.
# Summary of model 
The paper proposes a mixture model formulation of NMT where the mixing coefficients are uniform and fixed. The authors then proceed to derive a lowerbound on the marginal likelihood 
p(y|x) = \sum_z p(z)p(y|x,z) > 1/K \max_z p(y|x,z)
by picking the component z for which the joint likelihood is maximised. With a uniform p(z) this clearly selects the z for which the conditional p(y|x,z) is maximum. I use strictly greater here because p(z) > 0 and p(y|x,z) > 0 for every z.
The loss L(\theta|x,y) for an observation (x,y) is \min_z - \log p(y|z,x; \theta)
whose gradient with respect to NN parameters (theta) is \grad_theta \log p(y|z,x; \theta) for the component z that minimises the negative log-conditional and 0 for every other component, thus while this requires K forward passes (to solve \min_z), it only takes 1 backwards pass.
# Discussion
I appreciate model-based (as opposed to search-based) attempts to improve diversity for generation tasks such as MT. Latent variable modelling aims at a more explicit account of the generative procedure, namely, the joint distribution, which can potentially disentangle and explain different modes of the marginal. Thus from that point of view, this paper points to an exciting direction. That said, in my view, the assumptions behind the proposed approach are not justifiable and some of the claims are simply not appropriate. Below I try to support this view.
A stepping stone of this model is that p(y|x,z) must be "large for only one value of z" (as authors put it), and authors *assume* that will be the case. 
While the bound in equation (2) holds, whether or not p(y|z,x) turns out to be "large for only one value of z", it will be a very loose bound unless that happens. 
The key point is that one cannot *assume* it to be the case. One could perhaps *promote* it to be the case, but there's no aspect of the model formulation (or objective) that promotes such behaviour.
Backpropagating through whichever component happens to assign the largest likelihood does not guarantee (nor encourages) the other conditionals to *independently* end up going to zero. 
Given the level of parameter sharing, I'd even consider the possibility that the exact opposite happens. As authors put it themselves 
"Instead, by sharing parameters, even unpopular experts receive some gradients throughout training."
It's true they do, but they are being updated on the basis of the unilateral opinion of the selected component about the likelihood of the data.
Note that the true posterior p(z|x,y) is exactly proportional to the likelihood, as the prior is *uniform and fixed*:
  p(z|x,y) \propto 1/K p(y|x,z) \propto p(y|x,z)
This means that the authors expect the likelihood to do component allocation on its own. That is, the conditionals p(y|x,z=1), ..., p(y|x,z=K) must somehow coordinate themselves in making good use of the latent components. Without any mechanism to promote "competition" (in the parlance of Jacobs et al 1991), I don't see how this can work.
Also, the paper claims to model uncertainty, if I take the posterior to fulfil this claim, then I'm just left with a likelihood (again, due to uniform prior). In any case, a notion of uncertainty here would be conditioned on a point estimate of the network's parameters and should thus be worded carefully.
# Clarifications
1. "we aim to explicitly model uncertainty during training" can you make a case for where that happens in your model?
2. "prevents the gating from training well and the latent variable embeddings from specializing" which gating?
3. "While they showed improvements due to the regularization effect of the Monte Carlo gradient estimate”. I find it strange to talk about the “regularisation effect” of a gradient estimate, perhaps you can be a bit more precise here? Or perhaps you are referring to some specific component of the objective function whose gradient we are estimating via MC and perhaps that component may have some regularisation effect.
4. if you aim to have p(y|x,z) high for a single latent variable at a time, you are implicitly saying that every x has at most (or rather exactly) K translations with non-negligible probability. Is that sensible? 
# Pros/Cons
Pros
* simple: the approach presented here requires no significant changes to otherwise standard architectures, it instead concentrates in a change of objective and training algorithm.
* assessment of variability in translation: this paper proposes to use BLEU and a corpus of multiple references in an interesting (potentially novel) way. 
Cons
* problematic assumptions: e.g. posterior will turn out sparse without any explicit way to promote such behaviour
* unrealistic claims: e.g. modelling uncertainty
* imprecise use of technical language: some technical terms are not used in their strictly technical sense (e.g. uncertainty, degeneracy), some explanations employ loosely defined jargons (e.g. regularisation effect of the gradient estimate)