Neural text generation models such as recurrent networks are typically trained by maximizing data log-likelihood based on cross entropy. Such training objective shows a discrepancy from test criteria like the BLEU metric. Recent work optimizes expected BLEU under the model distribution using policy gradient, while such algorithm can suffer from high variance and become impractical. In this paper, we propose a new Differentiable Expected BLEU (DEBLEU) objective that permits direct optimization of neural generation models with gradient descent. We leverage the decomposability and sparsity of BLEU, and reformulate it with moderate approximations, making the evaluation of the objective and its gradient efficient, comparable to common cross-entropy loss. We further devise a simple training procedure with ground-truth masking and annealing for stable optimization. Experiments on neural machine translation and image captioning show our method significantly improves over both cross-entropy and policy gradient training.
Differentiable Expected BLEU for Text Generation
Paper Summary:
Neural translation systems optimizes training data likelihood, not the end metric of interest BLEU. This work proposes to approximate BLEU with a continuous, differentiable function that can be optimized during training.
Review:
The paper reads well. It has a few but crucial missing references. The motivation is easy to understand and a relevant problem to work on. The main weaknesses of the work lies in its very loose derivations, and its weak empirical results.
First on context/missing references: the author ignores approaches optimizing BLEU with log linear models (Franz Och 2003), and the structured prediction literature in general, both for exact (Tsochantaridis et al 2004) and approximate search (Daume and Marcu 2005). This type of approach has been applied to NMT recently (Edunov et al 2018). Your paper also misses important references addressing BLEU optimization with reinforcement strategies (Norouzi et al 2016) or (Bahdanau et al 2017). Although not targeting BLEU directly (Wiseman and Rush 16) is also a reference to cite wrt optimizing search quality directly. 
On empirical results, you chose to work IWSLT in the de-en direction while most of the literature worked on en-de. It prevents comparing your results to other papers. I would suggest to switch directions and/or to report results from other methods (Ranzato et al 2015; Wiseman and Rush 2016; Norouzi et al 2016; Edunov et al 2018). De-en is generally easier than en-de (generating German) and your BLEU scores are particularly low < 25 for de-en while other methods ranges in 26-33 BLEU for en-de (Edunov et al 2018).
On the method itself, approximating BLEU with a continuous function is not easy and the approach you take involves swapping function composition and expectation multiple times in a loose way. You acknowledge that (7) is unprincipled but (10) is also problematic since this equation does not acknowledge that successive ngrams overlap and cannot be considered independent. Also, the dependence of successive words is core to NMT/conditional language models and the independence hypothesis from the footnote on page 4 can be true only for a bag of word model. Overall, I feel that given the shortcuts you take, you need to justify that your approximation of BLEU is still correlated with BLEU. I would suggest to sample from a well trained NMT system to collect several hypotheses and to measure how well your BLEU approximation correlate with BLEU. How many times BLEU decides that hypA > hypB but your approximation invert this relation? is it true for large difference, small difference of BLEU score? at low BLEU score, high BLEU score?
Finally, you do not mention the distinction between expected BLEU  \sum_y P(y|x) BLEU(y, ref) and the BLEU obtained by beam search which only look at (an estimate of) the most likely sequence y* = argmax P(y|x) . Your approach and most reinforcement strategy targets optimizing expected BLEU, but this has no guarantee to make BLEU(y*, ref) any better. Could you report both an estimate of expected BLEU and beam BLEU for different methods? In particular, MERT (), beam optimization (Wiseman and Rush 2016) and  structured prediction (Edunov et al 2018) explicitly make this distinction. This is not a side issue as this discussion is in tension with your motivations.
Review Summary:
The paper misses important references. It chooses an empirical setup which prevents comparison with related work, and the report results on de-en seem weak. The proposed approach does not bound or estimate how far from BLEU is the proposed approximation. This means that the authors need to justify empirically that it preserves correlation with BLEU, which is not shown in the paper.
Missing references
An Actor-Critic Algorithm for Sequence Prediction (ICLR 2017)  Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, Yoshua Bengio
Hal Daume III and Daniel Marcu. Learning as search optimization: Approximate large margin methods for structured prediction. ICML 2005.
Sergey Edunov, Myle Ott, Michael Auli, David Grangier, Marc'Aurelio Ranzato
Classical Structured Prediction Losses for Sequence to Sequence Learning, NAACL 18
Minimum Error Rate Training in Statistical Machine Translation Franz Josef Och. 2003 ACL
I. Tsochantaridis, T. Hofmann, T. Joachims, and Y. Altun, Support Vector Machine Learning for Interdependent and Structured Output Spaces, ICML 2004.
Mohammad Norouzi, Samy Bengio, Zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, Dale Schuurmans, Reward Augmented Maximum Likelihood for Neural Structured Prediction, 2016
Sequence-to-Sequence Learning as Beam-Search Optimization, Sam Wiseman and Alexander M. Rush., EMNLP 2016