Neural sequence generation is commonly approached by using maximumlikelihood (ML) estimation or reinforcement learning (RL). However, it is known that they have their own shortcomings; ML presents training/testing discrepancy, whereas RL suffers from sample inefficiency. We point out that it is difficult to resolve all of the shortcomings simultaneously because of a tradeoff between ML and RL. In order to counteract these problems, we propose an objective function for sequence generation using α-divergence, which leads to an ML-RL integrated method that exploits better parts of ML and RL. We demonstrate that the proposed objective function generalizes ML and RL objective functions because it includes both as its special cases (ML corresponds to α → 0 and RL to α → 1). We provide a proposition stating that the difference between the RL objective function and the proposed one monotonically decreases with increasing α. Experimental results on machine translation tasks show that minimizing the proposed objective function achieves better sequence generation performance than ML-based methods.
This paper considers a dichitomy between ML and RL based methods for sequence generation. It is argued that the ML approach has some "discrepancy" between the optimization objective and the learning objective, and the RL approach suffers from bad sample complexity. An alpha-divergence formulation is considered to combine both methods.
Unfortunately, I do not understand main points made in this paper and am thus not able to give an accurate evaluation of the technical content of this paper. I therefore have no option but to vote for reject of this paper, based on my educated guess. 
Below are the points that I'm particularly confused about:
1. For the ML formulation, the paper made several particularly confusing remarks. Some of them are blatantly wrong to me. For example, 
1.1 The q(.|.) distribution in Eq. (1) *cannot* really be the true distribution, because the true distribution is unknown and therefore cannot be used to construct estimators. From the context, I guess the authors mean "empirical training distribution"?
1.2 I understand that the ML objective is different from what the users really care about (e.g., blue score), but this does not seem a "discrepancy" to me. The ML estimator simply finds a parameter that is the most consistent to the observed sequences; and if it fails to perform well in some other evaluation criterion such as blue score, it simply means the model is inadequate to describe the data given, or the model class is so large that the give number of samples is insufficient, and as a result one should change his/her modeling to make it more apt to describe the data at hand. In summary, I'm not convinced that the fact that ML optimizes a different objective than the blue score is a problem with the ML estimator.
In addition, I don't see at all why this discrepancy is a discrepancy between training and testing data. As long as both of them are identically distributed, then no discrepancy exists.
1.3 In point (ii) under the maximum likelihood section, I don't understand it at all and I think both sentences are wrong. First, the model is *not* trained on the true distribution which is unknown. The model is trained on an empirical distribution whose points are sampled from the true distribution. I also don't understand why it is evaluated using p_theta; if I understand correctly, the model is evaluated on a held-out test data, which is also generated from the underlying true distribution.
2. For the RL approach, I think it is very unclear as a formulation of an estimator. For example, in Eq. (2), what is r and what is y*? It is mentioned that r is a "reward" function, but I don't know what it means and the authors should perhaps explain further. I just don't see how one obtains an estimated parameter theta from the formulation in Eq. (2), using training examples.