What is percentage change in performance for better model when compared to baseline?
9.2% reduction in perplexity
Recent dialogue approaches operate by reading each word in a conversation history, and aggregating accrued dialogue information into a single state. This fixed-size vector is not expandable and must maintain a consistent format over time. Other recent approaches exploit an attention mechanism to extract useful information from past conversational utterances, but this introduces an increased computational complexity. In this work, we explore the use of the Neural Turing Machine (NTM) to provide a more permanent and flexible storage mechanism for maintaining dialogue coherence. Specifically, we introduce two separate dialogue architectures based on this NTM design. The first design features a sequence-to-sequence architecture with two separate NTM modules, one for each participant in the conversation. The second memory architecture incorporates a single NTM module, which stores parallel context information for both speakers. This second design also replaces the sequence-to-sequence architecture with a neural language model, to allow for longer context of the NTM and greater understanding of the dialogue history. We report perplexity performance for both models, and compare them to existing baselines.
Recently, chit-chat dialogue models have achieved improved performance in modelling a variety of conversational domains, including movie subtitles, Twitter chats and help forums BIBREF0, BIBREF1, BIBREF2, BIBREF3. These neural systems were used to model conversational dialogue via training on large chit-chat datasets such as the OpenSubtitles corpus, which contains generic dialogue conversations from movies BIBREF4. The datasets used do not have an explicit dialogue state to be modelled BIBREF5, but rather require the agent to learn the nuances of natural language in the context of casual peer-to-peer interaction. Many recent chit-chat systems BIBREF2, BIBREF3 attempt to introduce increased diversity into model responses. However, dialogue systems have also been known to suffer from a lack of coherence BIBREF0. Given an input message history, systems often have difficulty tracking important information such as professions and names BIBREF0. It would be of benefit to create a system which extracts relevant features from the input that indicate which responses would be most appropriate, and conditions on this stored information to select the appropriate response. A major problem with existing recurrent neural network (RNN) architectures is that these systems aggregate all input tokens into a state vector, which is passed to a decoder for generation of the final response, or in the case of a neural probabilistic language model BIBREF6, the state at each time step is used to predict the next token in the sequence. Ideally the size of the state should expand with the number of input tokens and should not lose important information about the input. However, RNN states are typically fixed sized, and for any chosen state size, there exists an input sequence length for which the RNN would not be able to store all relevant details for a final response. In addition, the RNN state undergoes constant transformation at each computational step. This makes it difficult to maintain a persistent storage of information that remains constant over many time steps. The introduction of attention mechanisms BIBREF7 has sparked a change in the current design of RNN architectures. Instead of relying fully on a fixed-sized state vector, an attention mechanism allows each decoder word prediction step to extract relevant information from past states through a key-value query mechanism. However, this mechanism connects every input token with all preceeding ones via a computational step, increasing the complexity of the calculation to $O(N^2)$ for an input sequence size N. In the ideal case, the mapping of input conversation history to output response would have a computational complexity of $O(N)$. For this reason, it is desirable to have an information retrieval system that is both scale-able, but not proportional to input length. We study the impact of accessible memory on response coherence by constructing a memory-augmented dialogue system. The motivation is that it would be beneficial to store details of the conversational history in a more permanent memory structure, instead of being captured inside a fixed-sized RNN hidden state. Our proposed system is able to both read and write to a persistent memory module after reading each input utterance. As such, it has access to a stable representation of the input message history when formulating a final response. We explore two distinct memory architectures with different properties, and compare their differences and benefits. We evaluate our proposed memory systems using perplexity evaluation, and compare them to competitive baselines.