Question generation is an important task for improving our ability to process natural language data, with additional challenges over other sequence transformation tasks. Recent approaches use modifications to a Seq2Seq architecture inspired by advances in machine translation, but unlike translation the input and output vocabularies overlap significantly, and there are many different valid questions for each input. Approaches using copy mechanisms and reinforcement learning have shown promising results, but there are ambiguities in the exact implementation that have not yet been investigated. We show that by removing inductive bias from the model and allowing the choice of generation path to become latent, we achieve substantial improvements over implementations biased with both naive and smart heuristics. We perform a human evaluation to confirm these findings. We show that although policy gradient methods may be used to decouple training from the ground truth and optimise directly for quality metrics that have previously been assumed to be good choices, these objectives are poorly aligned with human judgement and the model simply learns to exploit the weaknesses of the reward source. Finally, we show that an adversarial objective learned directly from the ground truth data is not able to generate a useful training signal.
The paper studies question generation, which is an important problem in many real applications. The authors propose to use better caching model and more evalution methods to deal with the problem. However, the paper is poorly written and hard to follow, and the proposed model lacks of novelty. The main reasons are as below:
1) In model section, the task definition is not clear. It is expected to see what's the question generation task studied in this paper. An example or a model overview will definitly help.
2) The encoder and decoder are not novel, it is expected to cite and compare with the existing similar encoder architecture, such as the encoder proposed in bidaf "Seo, Minjoon, et al. "Bidirectional attention flow for machine comprehension." arXiv preprint arXiv:1611.01603 (2016)."  The math symbols are aligned, for example, h_a or h^a is used to represent the encoding. Besides, adding the binary feature in the embedding is not necessary, the LSTM model could learn such sequential correlation. The decoder description is not clear as well and expected to compare with existing work (e.g., bidaf) to show the difference.
3) The proposed copy mecahnism is not clear. A formal definition of s_t, v_t and y_(t-1) should be given before defining the p_t. A more serious question, what is the fuse operation used to define p_t? concat, elementwise_plus or others?
4) In the training, how to deal the ground truth that are not in the vocab? The authors stated "using a modified heuristic described below", but no follow-ups in the paper.
5) The paper is not well written and organized. Small typos: in introduction, 'and and answer span', 'and output and output sequences'. In model, 'Glorot initialization', 'Bahdanau attention', it is not the common way to cite others' work. In encoder, the defintion of the state for decoder could be reorganized to the decoder.
I have read the authors' detailed rebuttal. Thanks.