How many hand-crafted templates did they have to make?
269
Question generation from a knowledge base (KB) is the task of generating questions related to the domain of the input KB. We propose a system for generating fluent and natural questions from a KB, which significantly reduces the human effort by leveraging massive web resources. In more detail, a seed question set is first generated by applying a small number of hand-crafted templates on the input KB, then more questions are retrieved by iteratively forming already obtained questions as search queries into a standard search engine, before finally questions are selected by estimating their fluency and domain relevance. Evaluated by human graders on 500 random-selected triples from Freebase, questions generated by our system are judged to be more fluent than those of \newcite{serban-EtAl:2016:P16-1} by human graders.
Question generation is important as questions are useful for student assessment or coaching purposes in educational or professional contexts, and a large-scale corpus of question and answer pairs is also critical to many NLP tasks including question answering, dialogue interaction and intelligent tutoring systems. There has been much literature so far BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 studying question generation from text. Recently people are becoming interested in question generation from KB, since large-scale KBs, such as Freebase BIBREF7 and DBPedia BIBREF8 , are freely available, and entities and their relations are already present in KBs but not for texts. Question generation from KB is challenging as function words and morphological forms for entities are abstracted away when a KB is created. To tackle this challenge, previous work BIBREF9 , BIBREF10 relies on massive human-labeled data. Treating question generation as a machine translation problem, serban-EtAl:2016:P16-1 train a neural machine translation (NMT) system with 10,000 $\langle $ triple, question $\rangle $ pairs. At test time, input triples are “translated” into questions with the NMT system. On the other hand, the question part of the 10,000 pairs are human generated, which requires a large amount of human effort. In addition, the grammaticality and naturalness of generated questions can not be guaranteed (as seen in Table 1 ). We propose a system for generating questions from KB that significantly reduces the human effort by leveraging the massive web resources. Given a KB, a small set of question templates are first hand-crafted based on the predicates in the KB. These templates consist of a transcription of the predicate in the KB (e.g. performsActivity $\Rightarrow $ how to) and placeholders for the subject (#X#) and the object (#Y#). A seed question set is then generated by applying the templates on the KB. The seed question set is further expanded through a search engine (e.g., Google, Bing), by iteratively forming each generated question as a search query to retrieve more related question candidates. Finally a selection step is applied by estimating the fluency and domain relevance of each question candidate. The only human labor in this work is the question template construction. Our system does not require a large number of templates because: (1) the iterative question expansion can produce a large number of questions even with a relatively small number of seed questions, as we see in the experiments, (2) multiple entities in the KB share the same predicates. Another advantage is that our system can easily generate updated questions as web is self-updating consistently. In our experiment, we compare with serban-EtAl:2016:P16-1 on 500 random selected triples from Freebase BIBREF7 . Evaluated by 3 human graders, questions generated by our system are significantly better then serban-EtAl:2016:P16-1 on grammaticality and naturalness.