How long of dialog history is captured?
two previous turns
In this work, we propose contextual language models that incorporate dialog level discourse information into language modeling. Previous works on contextual language model treat preceding utterances as a sequence of inputs, without considering dialog interactions. We design recurrent neural network (RNN) based contextual language models that specially track the interactions between speakers in a dialog. Experiment results on Switchboard Dialog Act Corpus show that the proposed model outperforms conventional single turn based RNN language model by 3.3% on perplexity. The proposed models also demonstrate advantageous performance over other competitive contextual language models.
Language model plays an important role in many natural language processing systems, such as in automatic speech recognition BIBREF0 , BIBREF1 and machine translation systems BIBREF2 , BIBREF3 . Recurrent neural network (RNN) based models BIBREF4 , BIBREF5 have recently shown success in language modeling, outperforming conventional n-gram based models. Long short-term memory BIBREF6 , BIBREF7 is a widely used RNN variant for language modeling due to its superior performance in capturing longer term dependencies. Conventional RNN based language model uses a hidden state to represent the summary of the preceding words in a sentence without considering context signals. Mikolov et al. proposed a context dependent RNN language model BIBREF8 by connecting a contextual vector to the RNN hidden state. This contextual vector is produced by applying Latent Dirichlet Allocation BIBREF9 on preceding text. Several other contextual language models were later proposed by using bag-of-word BIBREF10 and RNN methods BIBREF11 to learn larger context representation that beyond the target sentence. The previously proposed contextual language models treat preceding sentences as a sequence of inputs, and they are suitable for document level context modeling. In dialog modeling, however, dialog interactions between speakers play an important role. Modeling utterances in a dialog as a sequence of inputs might not well capture the pauses, turn-taking, and grounding phenomena BIBREF12 in a dialog. In this work, we propose contextual RNN language models that specially track the interactions between speakers. We expect such models to generate better representations of the dialog context. The remainder of the paper is organized as follows. In section 2, we introduce the background on contextual language modeling. In section 3, we describe the proposed dialog context language models. Section 4 discusses the evaluation procedures and results. Section 5 concludes the work.