Sequence generation models such as recurrent networks can be trained with a diverse set of learning algorithms. For example, maximum likelihood learning is simple and efficient, yet suffers from the exposure bias problem. Reinforcement learning like policy gradient addresses the problem but can have prohibitively poor exploration efficiency. A variety of other algorithms such as RAML, SPG, and data noising, have also been developed from different perspectives. This paper establishes a formal connection between these algorithms. We present a generalized entropy regularized policy optimization formulation, and show that the apparently divergent algorithms can all be reformulated as special instances of the framework, with the only difference being the configurations of reward function and a couple of hyperparameters. The unified interpretation offers a systematic view of the varying properties of exploration and learning efficiency. Besides, based on the framework, we present a new algorithm that dynamically interpolates among the existing algorithms for improved learning. Experiments on machine translation and text summarization demonstrate the superiority of the proposed algorithm.
The authors propose a more unified view of disparate methods for training sequence models. Specifically, a multi-term objective L(q,theta) consisting of:
1) The standard reward maximization objective of policy gradient, E_{p_\theta}[R], 
2) A weighted (weight alpha) reverse KL divergence of the parametric policy and a non-parameteric policy q, 
3) A weighted (weight beta) entropy term on q, 
Is proposed for sequence training (see equation (1). L can be iteratively optimized by solving for q given p_\theta, and the \theta given q (see eq. 2).
This framework mathematically generalizes softmax-policy gradient (SPG, alpha=1, beta=0), and reward-augmented maximum likelihood (alpha=0, beta=temperature), and also standard entropy regularized policy gradient (alpha=0), among other algorithms.
The paper is well written, and the approach sensible. However, combining SPG and RAML by introducing their respective regularization terms is a rather straightforward exercise, and so seems quite incremental.
Other major concerns are:
1) the true utility of the model, and 
2) the integrity of the experiments. 
Wrt: 
1), While RAML was a significant contribution at the time, it is now well established that RAML generally doesn't perform well at all in practice due to exposure bias (not conditioning on it's own previous predictions during training). Moreover SPG, as the authors point out, was supposed to address the need for ML pre-training, but required much engineering to work. The fact is that REINFORCE-based policy gradient methods are still more effective than these methods, provided they have a good baseline to reduce varince. Which brings me to point 
2) Was MIXER run with a learned baseline and judiciously optimized? Table 1 suggests that MIXER can outpeform ML by only 0.1 Bleu points, and outpeformed by RAML? Something is wrong with your implementation then. Moreover, there are techniques like self-critical sequence training (SCST), which far outpeform MIXER, and we haven't even discussed Actor-Critic baselines...
In summary, the contribution over RAML and SPG in combining them is quite incremental, and the practical importance of combining them is questionable, as is the integrity of the presented experiments, given how poorly MIXER is reported perform, and the omission of stronger baselines like SCST and AC methods. Also, a paper on essentially the same approach was submitted and rejected from ICLR 2018(https://openreview.net/pdf?id=H1Nyf7W0Z), although this paper is better written, and puts the method more fully in context with existing work, I think that several of the concerns with that paper apply here as well. 
Look forward to the authors' feedback, and additional/corrected results - will certainly update my score if these concerns are addressed. In particular, if this generalization can significantly outpeform existing methods it generalizes with non-degenerate settings, this would overcome the more incremental contribution of combining SPG and RAML.
Current Ratings:
Evaluation      2/5: Results are not consistent with previous results (e.g. MIXER results). Stronger baselines such as SCST and AC are not considered.
Clarity         5/5: Clear paper, well written.
Significance    3/5: RAML and SPG have not been established as important methods in practice, so combining them is less interesting.
Originality     2/5: RAML and SPG are fairly straightforward to combine for experts interested in these methods.
Rating          4/10 Okay but not good enough, reject.    
Confidence      5/5
Pros: 
- Generalizes RAML and SPG (and also standard entropy-regularized policy gradient).
- Well written paper, clean generalization.
Cons:
- RAML and SPG have not been established as important methods in practice.
- generalization of RAML and SPG is straightforward,  incremental.
- Existing baselines in the paper (i.e. MIXER) do not perform as expected (i.e barely better than ML, worse than RAML)
- Stronger REINFORCE-based algorthms like SCST, as well as Actor-critic algorithms, have not been compared.
Update after author responses:
--------------------------------------------
Authors, thank you for your feedback.
While it is true that generalizing RAML and SPG into a common framework is not trivial, the presented framework simply augments the dual form of SPG (i.e. REPS [16] in the SPG paper) with a RAML term. Furthermore, the MLE interpretation discussed is contained within the RAML paper, and the reductions to RAML and SPG are straightforward by design, and so do not really provide much new insight. Considering this, I feel that the importance of the paper largely rests on investigating and establishing the utility of the approach experimentally.
Wrt the experiments, I appreciate that the authors took the time to investigate the poor performance of MIXER. However, the unusally poor performance of MIXER remains unexplained, and falls short even of scheduled sampling (SS), which suggests a lingering major issue. REINFORCE techniques rely on 1) strong baselines, verified by the authors, 2) larger batch sizes to reduce variance, and 3) pre-training to reduce variance and facilitate efficient exploration. If the MLE is undertrained or overtrained (the latter the more likely issue given the plots), then MIXER will perform poorly. Actually, it is now standard practice to pre-train with MLE+SS before RL training, and this is really the (also dynamically weighted objective) baseline that should be compared against. The current REINFORCE results (MIXER or otherwise) really need to be updated (or at the least removed, as they are not captured by the framework, but the comparison to PG methods is important!).
More generally, I feel that the experiments are not yet comprehensive enough. While the authors have shown that they can outperform SPG and RAML with a scheduled objective, it is not currently clear how sensitive/robust the results are to the term weight scheduling, or even what most appropriate general weights/scheduling approach actually is.
Overall I feel that the paper is still in need of substantial maturation before publication, although I have revised my score slightly upward.