Spoken language understanding, usually including intent detection and slot filling, is a core component to build a spoken dialog system. Recent research shows promising results by jointly learning of those two tasks based on the fact that slot filling and intent detection are sharing semantic knowledge. Furthermore, attention mechanism boosts joint learning to achieve state-of-the-art results. However, current joint learning models ignore the following important facts: 1. Long-term slot context is not traced effectively, which is crucial for future slot filling. 2. Slot tagging and intent detection could be mutually rewarding, but bidirectional interaction between slot filling and intent detection remains seldom explored. In this paper, we propose a novel approach to model long-term slot context and to fully utilize the semantic correlation between slots and intents. We adopt a key-value memory network to model slot context dynamically and to track more important slot tags decoded before, which are then fed into our decoder for slot tagging. Furthermore, gated memory information is utilized to perform intent detection, mutually improving both tasks through global optimization. Experiments on benchmark ATIS and Snips datasets show that our model achieves state-of-the-art performance and outperforms other methods, especially for the slot filling task. Task-oriented dialogue systems have attracted sig-nificant attention, which have been greatly advanced by deep learning techniques. Traditionally, these dialog systems have been built as a pipeline, with modules including spoken language understanding (SLU), dialog state tracking, action selection and language generation. Among these problems, SLU, including intention detection and slot filling (Tur and Mori, 2011), is a key yet challenging problem to parse users' utterances into se-Sentence Flights from Irvine to Seattle Intent Flight Slots O O B-fromloc O B-toloc Table 1: An example utterance annotated with its intent and semantic slots (IOB format). mantic frames in order to capture a conversation's core meaning. Traditionally, intention detection is treated as a classification problem, whereas slot filling is usually defined as sequence labeling problem, where In-Out-Begin (IOB) format is applied for representing slot tags as illustrated in Table 1. Given an utterance, SLU determines users' intention and maps it into predefined semantic slots. The input is a sequence of words, and the output is a sequence of predefined slot IDs. A specific intent is assigned for the whole sentence. In the traditional pipeline approach, intent detection and slot filling are implemented separately. However, separate modeling of those two tasks is insufficient to take full advantage of all supervised signals, as they share semantic knowledge. For example, if the intent of an utterance is "find_a_flight", it is more likely to contain slots "de-parture_city" and "arrival_city" rather than "restau-rant_name". Another drawback of the pipeline method is that errors made in upper stream modules may propagate and be amplified in downstream components, which however could possibly be eased in joint model (Zhang and Wang, 2016). Recently, joint model for intent detection and slot filling has been proposed and achieved promising results (Liu and Lane, 2016; Goo et al., 2018; Li et al., 2018). Though achieving promising performance, their models suffer from two major issues: 1) Modeling of slot context. Though the latent memory of RNNs can model history information, they are inherently unstable over long time sequences because the memories are the RNN hidden states. (Weston et al., 2014) observes that RNNs tend to focus more on short-term memories and forcefully compress historical records into one hidden state vector. Thus, simple RNNs cannot preserve long-term slot context of the conversation, which is crucial to future slot tagging. 2) Bi-directional interaction between slot filling and intent detection. The majority of joint modeling work has studied how to utilize intent information to improve slot filling performance. However, the beneficial impact of slot information on intent detection is mostly ignored. In fact, slots and intents are closely correlative, thus mutually reinforcing each other. In this paper, we propose a new framework to jointly model intent detection and slot filling in order to achieve a deeper level of semantic modeling. Specifically, our model is distinguished from previous work primarily in two ways. Model slot context dynamically with Key-Value Memory Networks (KV-MNs). The majority of existing work use RNNs to track slot values mentioned in previous utterances. However, RNNs tend to focus more on short-term memories. We propose to use a memory network to model slot context information as external knowledge which is acting a global information to guide slot tagging. Instead of relying on the compressed vector in RNN, KV-MNs store different historical slot tag information separately in different memory slots, which enriches the representation capacity compared with RNNs. Furthermore, slot values mentioned in the utterance are dynamically tracked, which is beneficial for subsequent slot tagging at each timestamp. Lastly, slot-level attention can model more accurately the contribution of each word in an utterance to slot tagging. Model the mutual interaction between intent detection and slot filling. The fact that intent detection and slot filling are semantically related is well-observed and how to use intent information to boost slot filling is widely explored. However, slot filling is beneficial to intent detection as well, and these benefits are yet to be explored. We propose a gating mechanism between intents and slots based on KV-MNs in order to model the interaction between intent detection and slot filling. Since intent detection can be treated as an utterance classification problem, different classification methods, such as support vector machines (SVM) and RNNs (Haffner et al., 2003; Sarikaya et al., 2011), have proposed to solve it. On the other hand, for slot filling, hidden markov models (HMM) and conditional random fields (CRF) (Lee et al., 1992; Ye-Yi Wang et al., 2005; Raymond and Riccardi, 2007) were used to solve slot filling problem. Later RNN based methods had become popular. For example, Yao et al. (2013); Mesnil et al. (2015) employed RNNs for sequence labeling in order to perform slot filling. Alternatively, intent detection and slot filling can be done jointly to overcome the error propagation. Zhang and Wang (2016) first proposed joint work using RNNs for learning the correlation between intent and slots. Hakkani-Tr et al. (2016) adopted a RNN for slot filling and the last hidden state of the RNN was used to predict the utterance intent. Liu and Lane (2016) introduced an attention-based RNN encoder decoder model to jointly perform intent detection and slot filling. An attention weighted sum of all encoded hidden states was used to predict the utterance intent. All those models outperform the pipeline models via mutual enhancement between two tasks. Most recently, some work tries to model the intent information for slot filling explicitly in the joint model. Goo et al. (2018); Li et al. (2018) proposed the gate mechanism to explore incorporating the intent information for slot filling. However, as the sequence becomes longer, it is risky to simply rely on the gate function to sequentially summarize and compress all slots and context information in a single vector (Cheng et al., 2016). Wang et al. (2018) proposed the bi-model to consider the cross-impact between the intent and slots and achieve state-of-the-art results. Zhang et al. (2018) proposed a hierarchical capsule neural network to model the hierarchical relationship among word, slot, and intent in an utterance. Niu et al. (2019) introduces a SF-ID network to establish the interrelated mechanism for slot filling and intent detection tasks. Compared with their work, our method explicitly models the long-term slot context knowledge which is beneficial to both slot filling and intent detection. Memory network provides a principled approach for modeling long-range dependency which has advanced many NLP tasks such as machine translation (Wang et al., 2016) and question answering (Sukhbaatar et al., 2015). The initial framework of memory networks was proposed by Weston et al. (2014). Following the idea, Sukhbaatar et al. (2015) proposed an end-to-end memory augmented model that significantly reduced the requirement of supervision during training. Key-value memory network (Miller et al., 2016) encoded prior knowledge by introducing a key memory structure which storeed facts to address to the relevant memory value. None of them is to model slot context information dynamically especially in single turn conversational systems. In this paper, we demonstrate how memory networks can be used to model long-term slot context knowledge and the interaction between intent detection and slot filling. Memory networks show promising results on learning long-range dependency, but they are insensitive to represent temporal dependencies between memories (Wu et al., 2018). RNNs tend to be opposite. Thus, it makes sense for us to combine those networks together to model long-term slot context information. In this section, we present a specific key-value dynamic memory module to collect and remember slot clues in the dialog context. Then context memory is used to enhance the Encoder-Decoder based model to perform slot filling and intent detection. Given a single-turn dialog, the Encoder transforms a word in user utterances into a dense vector by using a shared self-attentive encoder. Then the memory network encodes long-term slot context information by incorporating historical slot tags through memory attention and WRITE operations of the memory network. The slot decoder integrates short-term hidden state of self-attention encoder and the long-term slot context generated by attentively reading the VALUE-MEMORY to generate slot tagging at each timestamp. Later, intent decoder performs token level intent detection, which is seen as a coarse-grained intent detection result. Finally, a fine-grained intent detection is produced by gating memory modules. Both intent detection and slot filling are optimized simultaneously via a joint learning scheme. Given an input utterance X = ( x 1 , x 2 , . . . , x T ) of T words, where each word is initially represented by a vector of dimension d , the BiLSTM (Hochre-iter and Schmidhuber, 1997) is applied to learn representations of each word by reading the input utterance forward and backward to produce context sensitive hidden states H = ( h 1 , h 2 , . . . , h T ) : Then, we use self-attention mechanism to capture the contextual information for each token. We adopt the method proposed by (Vaswani et al., 2017), where we first map the matrix of input vectors X  RT  d to queries ( Q ), keys (  K ) and values (  V ) matrices by using different linear projections and the self-attention output C  RT  d 1 is: C = softmax (cid:32) Q  K (cid:62)  d 2 (cid:33)  V (2) where d 1 and d 2 represents self-attention dimension and keys'dimension. We concatenate the output of self-attention and BiLSTM as the final encoding representation as shown in Qin et al. (2019): E = H  C (3) where E = ( e 1 , . . . , e T )  RT  ( d + d 1 ) and  is a concatenation operation. Our slot deocder consists of two components: 1) the key-value memory-augmented attention model which generates slot context representation of users' utterance, and 2) the unidirectional LSTM decoder, which predicts the next slot tag step by step. To overcome the shortcomings of RNNs in capturing semantic clues over the long-term, we design a memory network that can preserve fine-grained semantic information of long-term slot context. We adopt a key-value memory network, which memorizes information by using a large array of external memory slots. The external memories enrich the representation capability compared with hidden vectors of RNNs and enable the KV-MNs to capture long-term data characteristics (Liu and Perez, 2017). We aim to incorporate the knowledge contained in the historical slot tags into the memory slots. The KV-MNs decompose slot semantics in an utterance into different slot categories and thus preserves more fine-grained information. In KV-MNs, a memory slot is represented by a key vector and an associated value vector. KEY-MEMORY: The KEY-MEMORY K  R d k  n learns latent correlation between utterance words and slot tags, where n is the number of memory slots and d k is the dimension of each slot. Each column vector, that is, i -th key vector k i  R d k is set to the i th column of the KEY-MEMORY K , which is shared by all conversation turns and fixed during the processing of word sequences. VALUE-MEMORY: Both the KEY-MEMORY and VALUE-MEMORY have the same number of memory slots. Each value memory vector stores the value of slot tag mentioned in the utterance. We form a value memory matrix V t  R d v  n by combining all n value slots. Different from KEY-MEMORY K , VALUE-MEMORY V t is word-specific and is continuously updated according to the input word sequence. During the conversation, the value of a new slot tag may be added into the VALUE-MEMORY, and an old value can be erased. In this way, we can adequately capture the slot context information on each mentioned slot. Two types of operations, READ and WRITE , are designed to manipulate the value memories. As shown in Figure 1, the decoder uses the aligned BiLSTM hidden state h t as a query to address the KEY-MEMORY looking for an attention vector a t , and attentively reads the VALUE-MEMORY to generate slot context representation c t . First, we use h t to address the KEY-MEMORY to find an accurate attention vector a t . a t = Address ( h t , K ) (4) a t is subsequently used as the guidance for reading the VALUE-MEMORY V t  1 to get the slot context representation c t . c t = Read ( a t , V t  1 ) (5) c t works together with the aligned encoder hidden state e t to generate the new decoder state at the decoding step t , h St = LSTM (cid:0) h St  1 , y St  1 , e t  c t (cid:1) (6) where h St  1 is the previous slot decoder state and y St  1 is the previous emitted slot lable distribution. After that, we use the slot decoder hidden state h St to update V t : V t = Write (cid:0) h St , V t  1 (cid:1) (7) Finally, the decoder state h St is utilized for slot filling: y St = softmax (cid:0) W Sh h St (cid:1) (8) o St = argmax (cid:0) y St (cid:1) (9) where W Sh are trainable parameters and o St is the slot label of the word at timestamp t in the utterance. Different than most existing work where intent information is used to do slot filling, our framework is directly leveraging the explicit slot context information to help intent detection. Furthermore, a gated mechanism is used in order to effectively incorporate slot memory information into intent detection. By performing gated intent detection, there are two advantages: 1. Sharing slot context information with intent detection improves intent detection performance since those two tasks are related. Furthermore, a gating mechanism which combines the intent detection information and slot context retrieved from key-value memory, regulates the degree of enhancement of intent detection to prevent information overload. 2. Through shared key-value memory, the interaction between intent detection and slot filling can be effectively modeled and executed. Plus, by jointly training those two tasks, not only can intent detection performance be improved by slot context knowledge, but also slot filling is enhanced by minimizing intent detection objective function. In other words, by learning optimal parameters of shared key-value memory, slot filling and intent detection interact in a more effective and deeper way. Intent Detection Decoder: For intent detection, we use another uni-directional LSTM as the intent detection network. At each decode step t , the decoder state h It is generated by the previous decoder state h It  1 , the previous emitted intent label distribution y It  1 and the aligned encoder hidden e t . Gated Memory: We propose a gated mechanism to integrate slot context with intent detection. The gate regulates the degree of slot context information to feed into the intent detection task and prevent information from overloading. As shown in Figure 2, the gate G is a trainable fully connected network with sigmoid activation. Then, the output of gated decoder state h (cid:48) It is utilized for intent detection: y It = softmax (cid:16) W Ih h (cid:48) It (cid:17) (12) o It = argmax ( y It ) (13) where y It is the intent output distribution of the t -th token in the utterance, o It represents the intent lable of t -th token and W Ih are trainable parameters of the model. The final utterance result OI is generated by voting from all token intent results as illustrated in Qin et al. (2019). KEY-MEMORY Address: K  R d k  n denotes the KEY-MEMORY at decoding time step t . The addressed attention vector is given by a t = Address ( h t , K ) (14) where a t  R n specifies the normalized weights assigned to the slots in K , with j -th slot being k j . The attention weights a t,j are calculated based on the correlation between h t and k j : a t,j = exp ( e t,j ) (cid:80) ni =1 exp ( e t,i ) (15) where e t,j = k (cid:62) j ( W a h t + b a ) VALUE-MEMORY Read: V t  R d v  n denotes the VALUE-MEMORY at decoding time step t . The output of reading the value memory V t is given by c t = n (cid:88) j =1 a t,j v t,j (16) VALUE-MEMORY Write: Similar to the attentive writing operation of neural turing machines (Graves et al., 2014), we define two types of operation for updating the VALUE-MEMORY: FORGET and ADD. FORGET determines the content to be removed from memory slots. More specifically, the vector F t  R d v specifies the values to be forgotten or removed on each dimension in memory slots, which is then assigned to each memory slot through normalized weights a t . We use the slot decoder hidden state h St to update V t  1 . Formally, the memory after FORGET operation is given by  v t,i = v t  1 ,i ( 1  a t,i  F t ) , i = 1 , 2 , . . . , n (17) where  F t =  ( WF , h St ) is parameterized with WF  R d v  d h , and  stands for the Sigmoid activation function, and F t  R d v ;  a t  R n specifies the normalized weights assigned to the key memory slots in K , and a t,i represents the weight associated with the i -th memory slot. v t,i =  v t,i + a t,i  A t , i = 1 , 2 , . . . , n (18) where A t =  ( WA , h St ) is parameterized with WA  R d v  d h and A t  R d v . By learning the parameters of FORGET and ADD layers, our model can automatically determine which signal to weaken or strengthen based on input utterance words. L 1 (cid:44)  m (cid:88) j =1 n I (cid:88) i =1  y I,ij log (cid:16) y I,ij (cid:17) (19) and L 2 (cid:44)  m (cid:88) j =1 n S (cid:88) i =1  y S,ij log (cid:16) y S,ij (cid:17) (20) where  y I,ij and  y S,ij are the gold intent label and gold slot label respectively, m is the number of words in a word sequence, and n I and n S are the number of intent label types and the number of slot tag types, respectively. Finally the joint objective is formulated as weighted-sum of these two loss functions using hyper-parameters  and  : L  =  L 1 +  L 2 (21) Through joint training, the key-value memory shared by those two tasks can learn the shared representations and interactions between them, thus further promoting each other's performance and easing the error propagation compared with pipeline models. To evaluate our proposed model, we conduct experiments on two widely used benchmark datasets, ATIS (Airline Travel Information System) and Snips. Both datesets used in our paper follow the same format and partition as in Goo et al. (2018). ATIS dataset (Hemphill et al., 1990) contains audio recordings of people making flight reservations. The training set has 4,478 utterances and the test set contains 893 utterances. We use another 500 utterances for the development set. There are 120 slot labels and 21 intent types in the training sets. To justify the generalization of our proposed mode, we also execute our experiment on another NLU dataset collected by Snips (Coucke et al., 2018) 1 . This data is collected from the Snips personal voice assistant, where the number of samples for each intent is approximately the same. The training set contains 13,804 utterances and the test set contains 700 utterances. We use another 700 utterances as the development set. There are 72 slot labels and 7 intent types. Compared to single-domain ATIS dataset, Snips is more complicated mainly due to the intent diversity and large vocabulary (Goo et al., 2018). For example, GetWeather and BookRestaurant in Snips are from different top-ics, resulting in a larger vocabulary. On the other hand, intents in ATIS are all about flight information with similar vocabularies. In our experiments, we set the dimension of word embedding to 256 for ATIS and 200 for Snips dataset. L2 reularization used in our model is 1  10  6 and dropout ratio is set to 0.4 for reducing overfit. The number of memory columns is set to 20 for both datasets, and the dimensions of memory column vectors are set to 64 for ATIS, and to 200 for Snips. The optimizer is Adam (Kingma and Ba, 2014). During our experiments, we select the model which works the best on the development set, and then evaluate it on the test set. We carefully choose some representative works, for example, Joint Seq. (Hakkani-Tr et al., 2016), Attention BiRNN (Liu and Lane, 2016), Sloted-Gated (Goo et al., 2018), CAPSULE-NLU (Zhang et al., 2019), SF-ID Network (Niu et al., 2019) and Stack-Propagation (Qin et al., 2019) as our baselines. When doing the comparison, we adopt the reported results from those papers directly. In order to have fair comparison with others' work, we adopt the same metrics to evaluate our model. That is, we evaluate slot filling using F1 score, intent prediction using accuracy, and sentence-level semantic frame parsing using whole frame accuracy. Table 2 shows the experiment results of the proposed model on ATIS and Snips datasets. From the table, we can see that our model outperforms all the baselines in all three aspects: slot filling (F1), intent detection (Acc) and setence accurancy (Acc), demonstrating that explicitly modeling slot context and strong relationships between slots and intent can benefit SLU effectively from the key-value memory. In the ATIS dataset, compared with the best prior joint work Stack-Propagation (Qin et al., 2019), we achieve F1 score as 96.13 which is even slightly better than Stack-propagation's F1 score (96.10) with BERT model. This signifies that our key-value memory can not only capture long-term slot context, but also model correlation between slot filling and intent detection, which can be further optimized by joint training. What's more, in the Snips dataset, our model achieves good results in both slot filling and overall sentence. Specifically, slot filling was improved by almost 1.0%, and sentence accuracy by 1.4%. Generally, ATIS dataset is a simpler SLU task than Snips, and so the room to be improved is relatively small. On the other hand, Snips is more complex so that it needs more complicated model to capture long-term context and share the knowledge across different top-ics. In this section, we explore how each component contributes to our full model. Specifically, we ablate three important scenarios and conduct them in this experiment. Note that all the variants are based on joint learning. Without key-value memory and gating architecture for integrating slot context information with intent detection. This is the model similar to Qin et al. (2019). Only with key-value memory, but without sharing slot context information with intent detection. With key-value memory and sharing, but without gating architecture, where only key-value memory is applied to model slot context and that information is directly fed into intent detection. Table 3 shows the joint learning performance of our model on ATIS and Snips datasets by removing one component at one time. First, if we remove key-value memory and gating architecture, the performance drops dramatically compared with our proposed model. This is expected as it does not have any of our improvements. Then we only consider key-value memory to model slot context. From Table 3, we can see that key-value memory does improve performance in a large scale. The result can be interpreted as indicating that key-value memory learns long-term slot context representation effectively, which does compensate the weakness of RNN. In the following, we apply key-value memory and also share it with intent detection without gating. It is noticeable that SLU performance is enhanced further. Sharing slot context information with intent detection not only improves intent accuracy, but also betters slot filling through joint optimization. Finally, when we add gating mechanism, the performance improves further. We attribute this to gating mechanism that regulates the degree of slot context information to feed into intent detection task and prevent information from overloading. We also study how the number of memory slots and the dimension of memory slots impacts SLU performance. Figure 3 shows the performance change with different hyper-parameters. We found that the optimal size of memory slots for ATIS and Snips dataset is 20, whereas the optimal dimension of memory slots is 64 for ATIS and 200 for Snips respectively. Analyzing the attention weights has been frequently used to show the memory read-out, since it is an intuitive way to understand the model dynamics. Figure 4 shows the attention vector for each decoded slot, where each row represents attention vector a t . Our model has a sharp distribution over the memory, which implies that it is able to select the most related memory slots from the value memory. For example, when decoding "san", our model selects memory slot 1, 7, 8,15 from the value memory to read context information, where memory slot 7 and 15 are representing word "from" and memory slot 1 representing word "flight". In other words, words "flight" and "from" contribute more than other previous words in order to decode "san" to B-fromloc.city_name. 4: Key memory attention visualization from the ATIS dataset In this paper, we propose a joint model to perform spoken language understanding with an augmented key-value memory to model slot context in order to capture long-term slot information. In addition, we adopt a gating mechanism to incorporate slot context information for intent classification to improve intent detection performance. Reciprocally, joint optimization promotes slot filling performance further by memory sharing between those two tasks. Experiments on two public datasets show the effectiveness of our proposed model and achieve state-of-the-arts results.
abstain abstain abstain abstain abstain abstain objective method abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain objective objective objective objective objective objective objective objective objective objective objective other other other other other other other other other other other other other other other other method other other other other other objective abstain abstain method method abstain abstain abstain abstain abstain abstain abstain other method other abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain other method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain other abstain abstain abstain other method abstain abstain method other abstain abstain method other abstain abstain method method abstain abstain method abstain method abstain abstain method abstain abstain abstain abstain method method abstain abstain method method method method abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain objective result abstain objective