In this paper we introduce the Scratchpad Encoder, a novel addition to the sequence to sequence (seq2seq) framework and explore its effectiveness in generating natural language questions from a given logical form. The Scratchpad Encoder enables the decoder at each time step to modify all the encoder outputs, thus using the encoder as a "scratchpad" memory to keep track of what has been generated so far and to guide future generation. Experiments on a knowledge based question generation dataset show that our approach generates more fluent and expressive questions according to quantitative metrics and human judgments.
The paper studies the problem of question generation from sparql queries. The motivation is to generate more training data for knowledge base question answering systems to be trained on. However, this task is an instance of natural language generation: given a meaning representation (quite often a database record), generate the natural language text correspoding to it. And previous work on this topic has proposed very similar ideas to the scratchpad proposed here in order to keep track of what the neural decoder has already generated, here are two of them:
- Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-Hao Su, David Vandyke, Steve Young, EMNLP 2015: https://arxiv.org/abs/1508.01745
- Globally Coherent Text Generation with Neural Checklist Models
Chloe Kiddon Luke Zettlemoyer Yejin Choi: https://aclweb.org/anthology/D16-1032
Thus the main novelty claim of the paper needs to be hedged appropriately. Also, to demonstrate the superiority of the proposed method an appropriate comparison against previous work is needed.
Some other points:
- How is the linearization of the inout done? It  typically matters
- Given the small size of the dataset, I would propose experimenting with non-neural approaches as well, which are also quite common in NLG.
- On the human evaluation: showing the gold standard reference to the judges introduces bias to the evaluation which is inappropriate as in language generation tasks there are multiple correct answers. See this paper for discussion in the context of machine translation: http://www.aclweb.org/anthology/P16-2013
- For the automatic evaluation measures there should be multiple references per SPARQL query since this is how BLEU et al are supposed to be used. Also, this would allow to compare the references against each other (filling in the missing number in Table 4) and this would allow an evaluation of the evaluation itself: while perfect scores are unlikely, the human references should be much better than the systems.
- In the outputs shown in Table 3, the questions generated by the scratchpad encoder often seem to be too general compared to the gold standard, or incorrect. E.g. "what job did jefferson have" is semntically related to his role in the declaration of independence but rather different. SImilarly, being married to someone is not the same as having a baby with someone. While I could imagine human judges preferring them as they are fluent, I think they are wrong as they express a different meaning than the SPARQL query they are supposed to express. What were the guidelines used?