Does the model evaluated on NLG datasets or dialog datasets?
NLG datasets
Natural language generation (NLG) is a critical component in a spoken dialogue system. This paper presents a Recurrent Neural Network based Encoder-Decoder architecture, in which an LSTM-based decoder is introduced to select, aggregate semantic elements produced by an attention mechanism over the input elements, and to produce the required utterances. The proposed generator can be jointly trained both sentence planning and surface realization to produce natural language sentences. The proposed model was extensively evaluated on four different NLG datasets. The experimental results showed that the proposed generators not only consistently outperform the previous methods across all the NLG domains but also show an ability to generalize from a new, unseen domain and learn from multi-domain datasets.
Natural Language Generation (NLG) plays a critical role in Spoken Dialogue Systems (SDS) with task is to convert a meaning representation produced by the Dialogue Manager into natural language utterances. Conventional approaches still rely on comprehensive hand-tuning templates and rules requiring expert knowledge of linguistic representation, including rule-based BIBREF0 , corpus-based n-gram models BIBREF1 , and a trainable generator BIBREF2 . Recently, Recurrent Neural Networks (RNNs) based approaches have shown promising performance in tackling the NLG problems. The RNN-based models have been applied for NLG as a joint training model BIBREF3 , BIBREF4 and an end-to-end training model BIBREF5 . A recurring problem in such systems is requiring annotated datasets for particular dialogue acts (DAs). To ensure that the generated utterance representing the intended meaning of the given DA, the previous RNN-based models were further conditioned on a 1-hot vector representation of the DA. BIBREF3 introduced a heuristic gate to ensure that all the slot-value pair was accurately captured during generation. BIBREF4 subsequently proposed a Semantically Conditioned Long Short-term Memory generator (SC-LSTM) which jointly learned the DA gating signal and language model. More recently, Encoder-Decoder networks BIBREF6 , BIBREF7 , especially the attentional based models BIBREF8 , BIBREF9 have been explored to solve the NLG tasks. The Attentional RNN Encoder-Decoder BIBREF10 (ARED) based approaches have also shown improved performance on a variety of tasks, e.g., image captioning BIBREF11 , BIBREF12 , text summarization BIBREF13 , BIBREF14 . While the RNN-based generators with DA gating-vector can prevent the undesirable semantic repetitions, the ARED-based generators show signs of better adapting to a new domain. However, none of the models show significant advantage from out-of-domain data. To better analyze model generalization to an unseen, new domain as well as model leveraging the out-of-domain sources, we propose a new architecture which is an extension of the ARED model. In order to better select, aggregate and control the semantic information, a Refinement Adjustment LSTM-based component (RALSTM) is introduced to the decoder side. The proposed model can learn from unaligned data by jointly training the sentence planning and surface realization to produce natural language sentences. We conducted experiments on four different NLG domains and found that the proposed methods significantly outperformed the state-of-the-art methods regarding BLEU BIBREF15 and slot error rate ERR scores BIBREF4 . The results also showed that our generators could scale to new domains by leveraging the out-of-domain data. To sum up, we make three key contributions in this paper: We review related works in Section "Related Work" . Following a detail of proposed model in Section "Recurrent Neural Language Generator" , Section "Experiments" describes datasets, experimental setups, and evaluation metrics. The resulting analysis is presented in Section "Results and Analysis" . We conclude with a brief summary and future work in Section "Conclusion and Future Work" .