MINIMUM DIVERGENCE VS. MAXIMUM MARGIN: AN EMPIRICAL COMPARISON ON SEQ2SEQ MODELS
Sequence to sequence (seq2seq) models have become a popular framework for neural sequence prediction. While traditional seq2seq models are trained by Maximum Likelihood Estimation (MLE), much recent work has made various attempts to optimize evaluation scores directly to solve the mismatch between training and evaluation, since model predictions are usually evaluated by a task specific evaluation metric like BLEU or ROUGE scores instead of perplexity. This paper puts this existing work into two categories, a) minimum divergence, and b) maximum margin. We introduce a new training criterion based on the analysis of existing work, and empirically compare models in the two categories. Our experimental results show that our new training criterion can usually work better than existing methods, on both the tasks of machine translation and sentence summarization.
The authors have updated the paper and clarified some things, and now my impression of the paper has improved. It still feels a little incremental to me, but the potential application areas of these sorts of models are quite large and therefore incremental improvements are not insignificant. This paper suggests some natural follow-up work in exploring Hellinger distance and other variations for these models.
----- original review follows: ------
This paper discusses loss functions for sequence generation tasks that take into account cost functions that reflect the task-specific evaluation metric. They compare RAML and risk (MRT) formally and empirically, and also test a loss based on Hellinger distance. They compare these to some standard max-margin losses. MRT and the Hellinger distance loss perform best in NMT and summarization experiments. 
Pros:
There are some interesting aspects of this paper:
- It is interesting to note that RAML and MRT are (similar to) different directions of KL divergences between the same two distributions. (Caveat: the entropy regularizer, which I discuss in "Cons" below.)
- The new Hellinger-distance-based loss seems promising. 
- The empirical comparison among losses for standard NMT/summarization tasks is a potentially valuable contribution.
Cons:
A.
The focus/story of the paper need some work. It is unclear what the key contributions are. I think the Hellinger distance loss is potentially the most important contribution, but the authors don't spend much time on that.. it seems that they think the comparison of divergence and max-margin losses is more central. However, I think the authors' conclusion (that the divergence losses are better than the max-margin losses) is not the main story, because RAML is not much better than the max-margin losses. Also, I have some concerns about some of the details of the max-margin losses (listed and discussed below), so I'm not sure how reliable the empirical comparison is. 
B.
As for the connections and comparison between RAML and MRT: 
It does not seem that MRT corresponds to a divergence of the form given at the start of Sec. 4. There is also an entropy regularizer in Eq. (9). Sec. 4.3 states: "By comparing the above two methods, we find that both RAML and MRT are minimizing the KL divergence between the model output distribution and the exponentiated payoff distribution, but with different directions of D_KL." However, this statement ignores the entropy regularizer in Eq. (9). 
Maybe I'm being dense, but I didn't understand where Equation (10) comes from. I understand the equations above it for RAML, but I don't understand the MRT case in Eq. (10). Can you provide more details?
I also don't understand the following sentence: "It turns out that the hyperparameter \tau in RAML and \alpha in MRT have the same effect." What does this mean mathematically? Also, does this equivalence also require ignoring the entropy regularizer? As formulated, L_{RAML} necessarily contains a \tau, but L_{MRT} does not necessarily contain an alpha. It is only when moving to the sample approximation does the alpha become introduced. (MRT does not require this sample approximation; Some older work on MRT developed dynamic programming algorithms to exactly compute the gradients for structured output spaces like sequences, so samples were not used in those cases.) So I think the paper needs to clarify what exactly is meant by the connection between tau and alpha, under what conditions there is a connection between the two, and what exactly is the nature of this connection. If more space is needed for this, many of the details in Sec 3 can be cut or moved to appendices because those are standard and not needed for what follows. 
In the experimental results (Sec. 7), MRT outperforms RAML consistently. The authors discuss the impact of the directionality of the KL divergence, but what about the entropy regularizer? It would be interesting to compare MRT both with and without the entropy regularizer. Without the regularizer, MRT would actually correspond to the KL that the authors are describing in the discussion. As it currently stands, two things are changing between MRT and RAML and we don't know which is responsible for the sizable performance gains. 
C. 
There are several technical issues with the writing (and potentially with the claims/conclusions), most of which are potentially flexible with some corrections and more exposition by the authors:
Is L_{RAML} to be maximized or minimized? Looks like maximized, but clearly both L_{MLE} and L_{MRT} are supposed to be minimized, so the use of L for all of these seems confusing. If different conventions are to be used for each one, it should be explicitly mentioned in each case whether the term is to be maximized or minimized.
At the end of Sec. 4.1, q' is not defined. I can guess what it is, but it is not entirely clear from the context and should be defined. 
In Equation 6, please use different notation for the y in the denominator (e.g., y') to avoid collision with the y in the numerator and that on the left-hand side. 
The discussion of max-margin losses in Sec. 5 has some things that should be fixed. 
1. In Sec. 5, it is unclear why \Delta is defined to be the difference of two r() functions. Why not just make it -r(y, y^*)? Are there some implied conditions on \Delta that are not stated explicitly? If \Delta is assumed to be nonnegative, that should be stated. 
2. In Eq. (11), F appears to be a function of y and theta, but in the definition of F, it has no functional arguments. But then further down, F appears to be a function of y only. Please make these consistent.
3. Eq. (11) is not only hard for structured settings; it is also hard in the simplest settings (binary classification with 0-1 loss). This is the motivation for surrogate loss functions in empirical risk minimization for classification settings. The discussion in the paper makes it sound as if these challenges only arise in the structured prediction setting. 
4. I'm confused by what the paper is attempting to communicate with Equations 12 and 13. In Eq. 12, y on the left-hand side is not bound to anything, so it is unclear what is being stated exactly. Is it for all y? For any y? In Eq. 13, the \Delta on the right-hand side is outside the max over y -- is that really what was intended? I thought the max (the slack-rescaled loss-augmented inference step) should take into account \Delta. Otherwise, it is just doing an argmax over the score function. 
5. If the authors are directly optimizing the right-hand side of the inequality in Equation 12 (as would be suggested for the formula for the gradient), then there is no global minimum of the loss. It would go to negative infinity. Typically people use a "max(0, )" outside the loss so that the global minimum is 0. 
Typos and minor issues follow:
Sec. 1:
"the SEARN" --> "SEARN"
"the DAGGER" --> "DAGGER"
Sec. 2:
"genrally" --> "generally"
"took evaluation metric into training" --> "incorporated the evaluation metric into training"
"consistant" --> "consistent"
Sec. 3:
Use \exp instead of exp.
"a detail explanation" --> "a detailed explanation"
Sec. 4.1:
"predition" --> "prediction"
Sec. 4.2:
"only single sample" --> "only a single sample"
Sec. 6.1:
"less significant than" --> "less significantly than"