Neural Belief Tracker: Data-Driven Dialogue State Tracking
One of the core components of modern spoken dialogue systems is the belief tracker, which estimates the user’s goal at every step of the dialogue. However, most current approaches have difficulty scaling to larger, more complex dialogue domains. This is due to their dependency on either: a) Spoken Language Understanding models that require large amounts of annotated training data; or b) hand-crafted lexicons for capturing some of the linguistic variation in users’ language. We propose a novel Neural Belief Tracking (NBT) framework which overcomes these problems by building on recent advances in representation learning. NBT models reason over pre-trained word vectors, learning to compose them into distributed representations of user utterances and dialogue context. Our evaluation on two datasets shows that this approach surpasses past limitations, matching the performance of state-of-the-art models which rely on hand-crafted semantic lexicons and outperforming them when such lexicons are not provided.
1 Introduction
Spoken dialogue systems (SDS) allow users to interact with computer applications through conversation. Task-based dialogue systems help users achieve goals such as finding restaurants or booking flights. The dialogue state tracking (DST) component of an SDS serves to interpret user input and update the belief state, which is the system’s internal representation of the state of the conversation (Young et al., 2010). This is a probability distribution over dialogue states used by the downstream dialogue manager component
to decide which action the system should perform next. The Dialogue State Tracking Challenge (DSTC) series of shared tasks has provided a common evaluation framework accompanied by labelled datasets (Williams et al., 2016). In this framework, the dialogue system is supported by a domain ontology which describes the range of user intents the system can process. The ontology defines a collection of slots and the values each slot can take. The system must track the search constraints expressed by users (goals or informable slots) and questions the users ask about search results (requests), taking into account each user utterance (input via a speech recogniser) and the dialogue context (e.g., what the system just said). The following example shows the true state after each user utterance in a three-turn conversation:
User: I’m looking for a cheaper restaurant inform(price=cheap) System: How about Thai food? User: Yes please inform(price=cheap, food=Thai) System: The House serves cheap Thai food User: Where is it? inform(price=cheap, food=Thai); request(address) System: The House is at 106 Regent Street
DST models depend on identifying mentions of ontology items in user utterances, which becomes a non-trivial task when confronted with lexical variation, the dynamics of context and noisy speech recognition. Some approaches assume that a separate Spoken Language Understanding (SLU) module will solve this problem for them. However, such models require vast amounts of annotated training data. Moreover, coupling SLU and DST in a single model has been shown to improve belief tracking performance (Henderson et al., 2014d). On the other hand, such coupled models typically rely
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
FOOD=CHEAP: [affordable, budget, low-cost, low-priced, inexpensive, cheaper, economic, ...]
RATING=HIGH: [best, high-rated, highly rated, top-rated, cool, chic, popular, trendy, ...]
AREA=CENTRE: [center, downtown, central, city centre, midtown, town centre, ...]
Figure 1: An example semantic dictionary with rephrasings for three ontology values in a restaurant search domain.
on manually constructed semantic dictionaries to identify alternative mentions of ontology items that vary lexically or morphologically. Figure 1 gives an example of such a dictionary for three slot-value pairs. This approach, which we term delexicalisation, is clearly not scalable to larger, more complex dialogue domains.
In this paper, we present two new models, collectively called the Neural Belief Tracker (NBT) family. The proposed models couple SLU and DST, efficiently learning to handle variation without requiring any hand-crafted resources. To do that, NBT models move away from exact matching and instead reason entirely over pre-trained word vectors. The vectors making up the user utterance and preceding system output are first composed into intermediate representations. These are then used to decide which of the ontology-defined intents have been expressed by the user up to that point in the conversation.
To the best of our knowledge, NBT models are the first to successfully use pre-trained word vector spaces to improve the language understanding capability of belief tracking models. In evaluation on two datasets, we show that: a) NBT models match the performance of delexicalisation-based models which make use of hand-crafted semantic lexicons; and b) the NBT models significantly outperform those models when such resources are not available. Consequently, we believe this work proposes a framework better-suited to scaling belief tracking models for deployment in real-world dialogue systems operating over sophisticated application domains where the creation of such domain-specific lexicons would be infeasible.
2 Background
Models for probabilistic dialogue state tracking, or belief tracking, were introduced as components of spoken dialogue systems in order to better handle noisy speech recognition and other sources of uncertainty in understanding a user’s goals (Bohus
and Rudnicky, 2006; Williams and Young, 2007; Young et al., 2010). Modern dialogue management policies can learn to use a tracker’s distribution over intents to decide whether to execute an action or request clarification from the user. As mentioned above, the DSTC shared tasks have spurred research on this problem and established a standard evaluation paradigm (Williams et al., 2013; Henderson et al., 2014b,a). In this setting, the task is defined by an ontology that enumerates the goals a user can specify and the attributes of entities that the user can request information about. Many different belief tracking models have been proposed in the literature, from generative (Thomson and Young, 2010) and discriminative (Henderson et al., 2014d) statistical models to rule-based systems (Wang and Lemon, 2013). To motivate the work presented here, we categorise prior research according to their reliance (or otherwise) on a separate SLU module for interpreting user utterances:1
Separate SLU: Traditional SDS pipelines use Spoken Language Understanding (SLU) decoders to detect slot-value pairs expressed in the Automatic Speech Recognition (ASR) output. The downstream DST model then combines this information with the past dialogue context to update the belief state (Thomson and Young, 2010; Wang and Lemon, 2013; Lee and Kim, 2016; Perez, 2016; Perez and Liu, 2017; Sun et al., 2016; Jang et al., 2016; Shi et al., 2016; Dernoncourt et al., 2016; Liu and Perez, 2017; Vodolán et al., 2017). In the DSTC challenges, some systems used the output of template-based matching systems such as Phoenix (Wang, 1994). However, more robust and accurate statistical SLU systems are available. Many discriminative approaches to spoken dialogue SLU train independent binary models that decide whether each slot-value pair was expressed in the user utterance. Given enough data, these models can learn which lexical features are good indicators for a given value and can capture elements of paraphrasing (Mairesse et al., 2009). This line of work later shifted focus to robust handling of rich ASR output (Henderson et al., 2012; Tur et al., 2013). SLU has also been treated as a sequence labelling problem, where each word in an utterance is labelled according to its role in the user’s intent;
1The best-performing models in DSTC2 all used both raw ASR output and the output of SLU decoders (Williams, 2014; Williams et al., 2016). This does not mean that those models are immune to the drawbacks identified here for the two model categories; in fact, they share the drawbacks of both.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Figure 2: Architecture of the NBT Model. The implementation of the three representation learning subcomponents can be modified, as long as these produce adequate vector representations which the downstream model components can use to decide whether the current candidate slot-value pair was expressed in the user utterance (taking into account the preceding system act).
standard labelling models such as CRFs or Recurrent Neural Networks can then be used (Raymond and Ricardi, 2007; Yao et al., 2014; Celikyilmaz and Hakkani-Tur, 2015; Mesnil et al., 2015; Peng et al., 2015; Zhang and Wang, 2016; Liu and Lane, 2016b; Vu et al., 2016; Liu and Lane, 2016a, i.a.). Other approaches adopt a more complex modelling structure inspired by semantic parsing (Saleh et al., 2014; Vlachos and Clark, 2014). One drawback shared by these methods is their resource requirements, either because they need to learn independent parameters for each slot and value or because they need fine-grained manual annotation at the word level. This hinders scaling to larger, more realistic application domains.
Joint SLU/DST: Research on belief tracking has found it advantageous to reason about SLU and DST jointly, taking ASR predictions as input and generating belief states as output (Henderson et al., 2014d; Sun et al., 2014; Zilka and Jurcicek, 2015; Mrkšić et al., 2015). In DSTC2, systems which used no external SLU module outperformed all systems that only used external SLU features. Joint models typically rely on a strategy known as delexicalisation whereby slots and values mentioned in the text are replaced with generic labels. Once the dataset is transformed in this manner, one can extract a collection of template-like n-gram features such as [want tagged-value food]. To perform belief tracking, the shared model iterates over all slot-value pairs, extracting delexicalised feature vectors and making a separate binary decision regarding each pair. Delexicalisation introduces a hidden dependency that is rarely discussed: how do we identify slot/value mentions in text? For toy domains, one can manually construct semantic
dictionaries which list the potential rephrasings for all slot values. As shown by Mrkšić et al. (2016), the use of such dictionaries is essential for the performance of current delexicalisation-based models. Again though, this will not scale to the rich variety of user language or to general domains.
The primary motivation for the work presented in this paper is to overcome the limitations that affect previous belief tracking models. The NBT model efficiently learns from the available data by: 1) leveraging semantic information from pre-trained word vectors to resolve lexical/morphological ambiguity; 2) maximising the number of parameters shared across ontology values; and 3) having the flexibility to learn domainspecific paraphrasings and other kinds of variation that make it infeasible to rely on exact matching and delexicalisation as a robust strategy.
3 Neural Belief Tracker
The Neural Belief Tracker (NBT) is a model designed to detect the slot-value pairs that make up the user’s goal at a given turn during the flow of dialogue. Its input consists of the system dialogue acts preceding the user input, the user utterance itself, and a single candidate slot-value pair that it needs to make a decision about. For instance, the model might have to decide whether the goal FOOD=ITALIAN has been expressed in ‘I’m looking for good pizza’. To perform belief tracking, the NBT model iterates over all candidate slot-value pairs (defined by the ontology), and decides which ones have just been expressed by the user.
Figure 2 presents the flow of information in the model. The first layer in the NBT hierarchy performs representation learning given the three model
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
inputs, producing vector representations for the user utterance (r), the current candidate slot-value pair (c) and the system dialogue acts (tq, ts, tv). Subsequently, the learned vector representations interact through the context modelling and semantic decoding submodules to obtain the intermediate interaction summary vectors dr,dc and d. These are used as input to the final decision-making module which decides whether the user expressed the intent represented by the candidate slot-value pair.
3.1 Representation Learning
For any given user utterance, system act(s) and candidate slot-value pair, the representation learning submodules produce vector representations which act as input for the downstream components of the model. All representation learning subcomponents make use of pre-trained collections of word vectors. As shown by Mrkšić et al. (2016), specialising word vectors to express semantic similarity rather than relatedness is essential for improving belief tracking performance. For this reason, we use the semantically-specialised Paragram-SL999 word vectors (Wieting et al., 2015) throughout this work. The NBT training procedure keeps these vectors fixed: that way, at test time, unseen words semantically related to familiar slot values (i.e. inexpensive to cheap) will be recognised purely by their position in the original vector space (see also Rocktäschel et al. (2016)). This means that the NBT model parameters can be shared across all values of the given slot, or even across all slots.
Let u represent a user utterance consisting of ku words u1, u2, . . . , uku . Each word has an associated word vector u1, . . . ,uku . We propose two model variants which differ in the method used to produce vector representations of u: NBT-DNN and NBT-CNN. Both act over the constituent ngrams of the utterance. Let vni be the concatenation of the n word vectors starting at index i, so that:
vni = ui ⊕ . . .⊕ ui+n−1 (1)
where⊕ denotes vector concatenation. The simpler of our two models, which we term NBT-DNN, is shown in Figure 3. This model computes cumulative n-gram representation vectors r1, r2 and r3, which are the n-gram ‘summaries’ of the unigrams, bigrams and trigrams in the user utterance:
rn = ku−n+1∑ i=1 vni (2)
Each of these vectors is then non-linearly mapped to intermediate representations of the same size:
r′n = σ(W s nrn + b s n) (3)
where the weight matrices and bias terms map the cumulative n-grams to vectors of the same dimensionality and σ denotes the sigmoid activation function. We maintain a separate set of parameters for each slot (indicated by superscript s). The three vectors are then summed to obtain a single representation for the user utterance:
r = r′1 + r ′ 2 + r ′ 3 (4)
The cumulative n-gram representations used by this model are just unweighted sums of all word vectors in the utterance. Ideally, the model should learn to recognise which parts of the utterance are more relevant for the subsequent classification task. For instance, it could learn to ignore verbs or stop words and pay more attention to adjectives and nouns which are more likely to express slot values.
Our second model, NBT-CNN, draws inspiration from successful applications of Convolutional Neural Networks (CNNs) for language understanding (Collobert et al., 2011; Kalchbrenner et al., 2014; Kim, 2014). These models typically apply a number of convolutional filters to n-grams in the input sentence, followed by non-linear activation functions and max-pooling. Following this approach, the NBT-CNN model applies L = 300 different filters for n-gram lengths of 1, 2 and 3 (Figure 4). Let F sn ∈ RL×nD denote the collection of filters for each value of n, where D = 300 is the word vector dimensionality. If vni denotes the concatenation of n word vectors starting at index i, let mn = [vn1 ;v n 2 ; . . . ;v n ku−n+1] be the list of n-grams that convolutional filters of length n run over. The three intermediate representations are then given by:
Rn = F s n mn (5)
Each column of the intermediate matrices Rn is produced by a single convolutional filter of length n. We obtain summary n-gram representations by pushing these representations through a rectified linear unit (ReLu) activation function (Nair and Hinton, 2010) and max-pooling over time (i.e. columns of the matrix) to get a single feature for each of the L filters applied to the utterance:
r′n = maxpool (ReLu (Rn + b s n)) (6)
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Figure 3: NBT-DNN Model. Word vectors of all unigrams, bigrams and trigrams are summed to obtain cumulative n-gram representations. These are passed through another hidden layer and then summed to obtain the utterance representation r.
Figure 4: NBT-CNN Model. L convolutional filters of window sizes 1, 2, 3 are applied to word vectors of the given utterance (L = 3 in the diagram, but L = 300 in the system). The convolutions are followed by the ReLu activation function and max-pooling to produce summary n-gram representations. These are summed to obtain the utterance representation r.
where bsn is a bias term broadcast across all filters. Finally, the three summary n-gram representations are summed to obtain the final utterance representation vector r (as in Equation 4). The NBT-CNN model is (by design) better suited to longer utterances, as its convolutional filters interact directly with subsequences of the utterance, and not just their noisy summaries given by the NBT-DNN’s cumulative n-grams.
3.2 Semantic Decoding
The NBT diagram in Figure 2 shows that the utterance representation r and the candidate slotvalue pair representation c directly interact through the semantic decoding module. This component decides whether the user explicitly expressed an intent matching the current candidate pair (i.e. without taking the dialogue context into account). Examples of such matches would be ‘I want Thai food’ with food=Thai or more demanding ones such as ‘a pricey restaurant’ with price=expensive. This is where the use of high-quality pre-trained word vectors comes into play: a delexicalisation-based model could deal with the former example but would be helpless in the latter case, unless a human expert had provided a semantic dictionary listing all potential rephrasings for each value in the domain ontology.
Let the vector space representations of a candidate pair’s slot name and value be given by cs and cv (with vectors of multi-word slot names/values summed together). The NBT model learns to map this tuple into a single vector c of the same dimensionality as the utterance representation r. These two representations are then forced to interact in order to learn a similarity metric which discriminates between interactions of utterances with slot-value pairs that they either do or do not express:
c = σ (W sc (cs + cv) + b s c) (7) d = r⊗ c (8)
where ⊗ denotes element-wise vector multiplication. The dot product, which may seem like the more intuitive similarity metric, would reduce the rich set of features in d to a single scalar. The element-wise multiplication allows the downstream network to make better use of its parameters by learning non-linear interactions between sets of features in r and c.2 The downstream network (Binary Decision Maker in Figure 2) uses one intermediate hidden layer of size 100 to make the final binary decision regarding the current candidate pair.
2We also tried to concatenate r and c and pass that vector to the downstream decision-making neural network. However, this set-up led to very weak performance since our relatively small datasets did not suffice for the network to learn to model the interaction between the two feature vectors.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
3.3 Context Modelling
This ‘decoder’ does not yet suffice to extract intents from utterances in human-machine dialogue. To understand some queries, the belief tracker must be aware of context, i.e. the flow of dialogue leading up to the latest user utterance. While all previous system and user utterances are important, the most relevant one is the last system utterance, in which the dialogue system could have performed (among others) one of the following two system acts:
1. System Request: The system asks the user about the value of a specific slot Tq. If the system utterance is: ‘what price range would you like?’ and the user answers with any, the model must infer the reference to price range, and not to other slots such as area or food.
2. System Confirm: The system asks the user to confirm whether a specific slot-value pair (Ts, Tv) is part of their desired constraints. For example, if the user responds to ‘how about Turkish food?’ with ‘yes’, the model must be aware of the system act in order to correctly update the belief state.
If we make the Markovian decision to only consider the last set of system acts, we can incorporate context modelling into the NBT model. Let tq and (ts, tv) be the word vector representations of the arguments for the system request and confirm acts (zero vectors if none). The model computes the following measures of similarity between the system acts, candidate pair (cs, cv) and utterance representation r:
dr = (cs · tq)r (9) dc = (cs · ts)(cv · tv)r (10)
where · denotes dot product. The computed similarity terms act as gating mechanisms which only pass the utterance representation through if the system asked about the current candidate slot or slot-value pair. This type of interaction is particularly useful for the confirm system act: if the system asks the user to confirm, the user is likely not to mention any slot values, but to just respond affirmatively or negatively. This means that the model must consider the three-way interaction between the utterance, candidate slot-value pair and the slot value pair offered by the system. If (and only if) the latter two are the same should the model consider the affirmative or negative polarity of the user utterance when
making the subsequent binary decision. Finally, these two context modelling summary representations are passed to the decision making module, which combines them with the semantic decoder output d to make the final binary decision.
4 Belief State Update Mechanism
In spoken dialogue systems, belief tracking models operate over the output of automatic speech recognition (ASR). Despite improvements to speech recognition, the need to make the most out of imperfect ASR will persist as dialogue systems are used in increasingly noisy environments.
In this work, we define a simple rule-based belief state update mechanism which can be applied to ASR N -best lists. For dialogue turn t, let syst−1 denote the preceding system output, and let ht denote the list of N ASR hypotheses hti with posterior probabilities pti. For any hypothesis h t i, slot s and slot value v ∈ Vs, NBT models estimate P(s, v | hti, syst−1), which is the (turn-level) probability that (s, v) was expressed in the given hypothesis. The predictions for N such hypotheses are then combined as:
P(s, v | ht, syst−1) = N∑ i=1 pti P ( s, v | hti, syst ) This turn-level belief state estimate is then combined with the (cumulative) belief state up to time (t− 1) to get the updated belief state estimate: P(s, v | h1:t, sys1:t−1) = λ P ( s, v | ht, syst−1
) + (1− λ) P ( s, v | h1:t−1, sys1:t−2
) where λ is the coefficient which determines the relative weight of the turn-level and previous turns’ belief state estimates.3 For slot s, the set of its detected values at turn t is then given by:
V ts = {v ∈ Vs | P ( s, v | h1:t, sys1:t−1 ) ≥ 0.5}
For informable (i.e. goal-tracking) slots, the value in V ts with the highest probability is chosen as the current goal (if V ts 6= {∅}). For requests, all slots in V treq are deemed to have been requested. As requestable slots serve to model single-turn user queries, they require no belief tracking across turns.
3This coefficient was tuned on the DSTC2 development set. The best performance was achieved with λ = 0.55.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
5 Experiments
5.1 Datasets
Two datasets were used for training and evaluation. Both consist of user conversations with taskoriented dialogue systems designed to help users find suitable restaurants around Cambridge, UK. The two corpora share the same domain ontology, which contains three informable (i.e. goal-tracking) slots: FOOD TYPE, AREA and PRICE. The users can specify values for these slots in order to find restaurants which best meet their criteria. Once the system suggests a restaurant, the users can ask about the values of up to eight requestable slots (PHONE NUMBER, ADDRESS, etc.). The two datasets are:
1. DSTC2: We use the transcriptions, ASR hypotheses and turn-level semantic labels provided for the Dialogue State Tracking Challenge 2 (Henderson et al., 2014a). The official transcriptions contain various spelling errors which we corrected manually; the cleaned version of the dataset will be made available on the first author’s website. The training data contains 2207 dialogues and the test set consists of 1117 dialogues. We train NBT models on transcriptions and report belief tracking performance on test set ASR predictions.
2. WOZ 2.0: Wen et al. (2017) performed a Wizard of Oz style experiment in which Amazon Mechanical Turk users assumed the role of the system or the user of a task-oriented dialogue system based on the DSTC2 ontology. Users typed instead of using speech, which means performance in the WOZ experiments is more indicative of the model’s capacity for semantic understanding than its robustness to ASR errors. Whereas in the DSTC2 dialogues users would quickly adapt to the system’s (lack of) language understanding capability, the WOZ experimental design gave them freedom to use more sophisticated language. We expanded the original WOZ dataset from Wen et al. (2017) using the same data collection procedure, yielding a total of 1200 dialogues (5012 turns). We divided these into 600 training, 200 validation and 400 test set dialogues. The WOZ 2.0 dataset will also be available from the first author’s website.
We focus on two key evaluation metrics introduced in (Henderson et al., 2014a):
1. Goals (‘joint goal accuracy’): the proportion of dialogue turns where all the user’s search goal constraints were correctly identified;
2. Requests: similarly, the proportion of dialogue turns where user’s requests for information were identified correctly.
5.2 Models
We evaluate two NBT model variants: NBT-DNN and NBT-CNN. The two corpora are used to create training data for two separate experiments. We iterate over all utterances, generating one example for each of the slot-value pairs in the ontology. An example consists of a transcription, its context (i.e. list of preceding system acts) and a candidate slot-value pair. The binary label for each example indicates whether or not its utterance and context express the example’s candidate pair. To train the models, we used the Adam optimizer (Kingma and Ba, 2015) with cross-entropy loss, backpropagating through all the NBT subcomponents while keeping the pre-trained word vectors fixed.
For each dataset, we compare the NBT models to: 1) a baseline system that implements a wellknown competitive delexicalisation-based model for that dataset; and 2) the same baseline model supplemented with a task-specific semantic dictionary (produced by the baseline system creators). For DSTC2, the model is that of Henderson et al. (2014d; 2014c). This model is an n-gram based neural network model with recurrent connections between turns (but not inside utterances) which replaces occurrences of slot names and values with generic delexicalised features. For WOZ 2.0, we compare the NBT models to a more sophisticated belief tracking model presented in (Wen et al., 2017). This model uses an RNN for belief state updates and a CNN for turn-level feature extraction. Unlike NBT-CNN, their CNN operates not over vectors, but over delexicalised features akin to those used by Henderson et al. (2014c).
Both baseline models map exact matches of ontology-defined intents (and their lexiconspecified rephrasings) to one-hot delexicalised ngram features. This means that pre-trained vectors cannot be incorporated directly into these models.
6 Results
6.1 Belief Tracking Performance
Table 1 shows the performance of NBT models trained and evaluated on DSTC2 and WOZ 2.0 datasets. The NBT models outperformed the baseline models in terms of both joint goal and request accuracies. For goals, the gains are always statis-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Model DSTC2 WOZ 2.0Goals Requests Goals Requests Baseline DST 69.1 95.7 70.8 87.1 + sem. dict. 72.9* 95.7 83.7* 87.6 NBT-DNN 72.6* 96.4 84.4* 91.2* NBT-CNN 73.4* 96.5 84.2* 91.6*
Table 1: DSTC2 and WOZ 2.0 test set accuracies for: a) joint goals; and b) turn-level requests. The asterisk indicates statistically significant improvement over the baseline delexicalisation-based trackers (paired t-test; p < 0.05).
tically significant (paired t-test, p < 0.05). Moreover, there was no statistically significant variation between the NBT and the lexicon-supplemented models, showing that the NBT can handle semantic relations which otherwise had to be explicitly encoded in semantic dictionaries.
While the NBT performs well across the board, we can compare its performance on the two datasets to understand its strengths. The improvement over the baseline is greater on WOZ 2.0, which corroborates our intuition that the NBT’s ability to learn linguistic variation is vital for this dataset containing longer sentences, richer vocabulary and no ASR errors. By comparison, the language of the subjects in the DSTC2 dataset is less rich, and compensating for ASR errors is the main hurdle: given access to the DSTC2 test set transcriptions, the NBT models’ goal accuracy rises to 0.96. This indicates that future work should focus on better ASR compensation if the model is to be deployed in environments with challenging acoustics.
6.2 The Importance of Word Vector Spaces
The NBT models use the semantic relations embedded in the pre-trained word vectors to handle semantic variation and produce high-quality intermediate representations. Table 2 shows the performance of NBT-CNN4 models making use of three different word vector collections: 1) ‘random’ word vectors initialised using the XAVIER initialisation (Glorot and Bengio, 2010); 2) distributional GloVe vectors (Pennington et al., 2014), trained using co-occurrence information in large textual corpora; and 3) semantically specialised ParagramSL999 vectors (Wieting et al., 2015), which are obtained by injecting semantic similarity constraints from the Paraphrase Database (Ganitkevitch et al., 2013) into distributional word vectors in order to improve their semantic content.
The results in Table 2 show that the use of seman4The NBT-DNN model showed the same trends. For
brevity, Table 2 presents only the NBT-CNN figures.
tically specialised word vectors leads to considerable performance gains: Paragram-SL999 vectors (significantly) outperformed GloVe and XAVIER vectors for goal tracking on both datasets. The gains are particularly robust for noisy DSTC2 data, where both collections of pre-trained vectors consistently outperformed random initialisation. The gains are weaker for the noise-free WOZ 2.0 dataset, which seems to be large (and clean) enough for the NBT model to learn task-specific rephrasings and compensate for the lack of semantic content in the word vectors. For this dataset, GloVe vectors do not improve over the randomly initialised ones. We believe this happens because distributional models keep related, yet antonymous words close together (e.g. north and south, expensive and inexpensive), offsetting the useful semantic content embedded in this vector spaces.
7 Conclusion
In this paper, we have proposed a novel neural belief tracking (NBT) framework designed to overcome current obstacles to deploying dialogue systems in real-world dialogue domains. The NBT models offer the known advantages of coupling Spoken Language Understanding and Dialogue State Tracking, without relying on hand-crafted semantic lexicons to achieve state-of-the-art performance. Our evaluation demonstrated these benefits: the NBT models match the performance of models which make use of such lexicons and vastly outperform them when these are not available. Finally, we have shown that the performance of NBT models improves with the semantic quality of the underlying word vectors. To the best of our knowledge, we are the first to move past intrinsic evaluation and show that semantic specialisation boosts performance in downstream tasks.
In future work, we intend to explore applications of the NBT for multi-domain dialogue systems, as well as in languages other than English that require handling of complex morphological variation.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This paper presents a neural network-based framework for dialogue state tracking. The main contribution of this work is on learning representations of user utterances, system outputs, and also ontology entries, all of which are based on pre-trained word vectors. Particularly for the utterance representation, the authors compared two different neural network models: NBT-DNN and NBT-CNN. The learned representations are combined with each other and finally used in the downstream network to make binary decision for a given slot value pair. The experiment shows that the proposed framework achieved significant performance improvements compared to the baseline with the delexicalized approach.  It's generally a quality work with clear goal, reasonable idea, and improved results from previous studies. But the paper itself doesn't seem to be very well organized to effectively deliver the details especially to readers who are not familiar with this area.  First of all, more formal definition of DST needs to be given at the beginning of this paper. It is not clear enough and could be more confusing after coupling with SLU. My suggestion is to provide a general architecture of dialogue system described in Section 1 rather than Section 2, followed by the problem definition of DST focusing on its relationships to other components including ASR, SLU, and policy learning.  And it would also help to improve the readability if all the notations used throughout the paper are defined in an earlier section. Some symbols (e.g. t_q, t_s, t_v) are used much earlier than their descriptions.  Below are other comments or questions:  - Would it be possible to perform the separate SLU with this model? If no, the term 'joint' could be misleading that this model is able to handle both tasks.  - Could you please provide some statistics about how many errors were corrected from the original DSTC2 dataset? If it is not very huge, the experiment could include the comparisons also with other published work including DSTC2 entries using the same dataset.  - What do you think about using RNNs or LSTMs to learn the sequential aspects in learning utterance representations? Considering the recent successes of these recurrent networks in SLU problems, it could be effective to DST as well.  - Some more details about the semantic dictionary used with the baseline would help to imply the cost for building this kind of resources manually.  - It would be great if you could give some samples which were not correctly predicted by the baseline but solved with your proposed models.


Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots
We study response selection for multi-turn conversation in retrieval based chatbots. Existing work either concatenates utterances in context or matches a response with a highly abstract context vector finally, which may lose relationships among the utterances or important information in the context. We propose a sequential matching network (SMN) to address both problems. SMN first matches a response with each utterance in the context on multiple levels of granularity, and distills important matching information from each pair as a vector with convolution and pooling operations. The vectors are then accumulated in a chronological order through a recurrent neural network (RNN) which models relationships among the utterances. The final matching score is calculated with the hidden states of the RNN. Empirical study on two public data sets shows that SMN can significantly outperform state-of-the-art methods for response selection in multi-turn conversation.
1 Introduction
Conversational agents include task-oriented dialog systems and non-task-oriented chatbots. Dialog systems focus on helping people complete specific tasks in vertical domains (Young et al., 2010), while chatbots aim to naturally and meaningfully converse with humans on open domain topics (Ritter et al., 2011). Existing work on building chatbots includes generation based methods and retrieval based methods. Retrieval based chatbots enjoy the advantage of informative and fluent responses, because they select a proper response for the current conversation from a repository with re-
sponse selection algorithms. While most existing work on retrieval based chatbots studies response selection for single-turn conversation (Wang et al., 2013) which only considers the last input message, we consider the problem in a multi-turn scenario. In a chatbot, multi-turn response selection takes a message and utterances in its previous turns as input and selects a response that is natural and relevant to the whole context.
The key to response selection lies in inputresponse matching. Different from single-turn conversation, multi-turn conversation requires matching between a response and a conversation context in which one needs to consider not only the matching between the response and the input message but also the matching between the response and the utterances in previous turns. The challenges of the task include (1) how to identify important information (words, phrases, and sentences) in the context that is crucial to selecting a proper response and how to leverage the information in matching; and (2) how to model relationships among the utterances in the context. Table 1 illustrates the challenges with an example. First, “hold a drum class” and “drum” in the context are very important. Without them, one may find responses relevant to the message (i.e., the last turn of the context) but nonsense in the context (e.g., “what lessons do you want?”). Second,
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
the message highly depends on the second turn in the context, and the order of the utterances matters in response selection: exchanging the third turn and the last turn may lead to different responses. Existing work, however, either ignores relationships among utterances when concatenating them together (Lowe et al., 2015), or loses important information in context in the process of converting the whole context to a vector without enough supervision from responses (e.g., by a hierarchical RNN (Zhou et al., 2016)).
We propose a sequential matching network (SMN), a new context based matching model that can tackle both challenges in an end-to-end way. The reason that existing models lose important information in the context is that they first represent the whole context as a vector and then match the context vector with a response vector. Thus responses in these models cannot meet the context until the final step in matching. To avoid information loss, SMN matches a response with each utterance in the context at the beginning and encodes important information in each pair into a matching vector. The matching vectors are then accumulated in the temporal order of the utterances to model their relationships. The final matching degree is computed with the accumulation of the matching vectors. Specifically, for each utteranceresponse pair, the model constructs a word-word similarity matrix and a sequence-sequence similarity matrix by the embedding of words and the hidden states of a recurrent neural network with gated recurrent unites (GRU) (Chung et al., 2014) respectively. The two matrices capture important matching information in the pair on a word level and a segment level respectively, and the information is distilled and fused as a matching vector through an alternation of convolution and pooling operations on the matrices. By this means, important information from multiple levels of granularity in the context is recognized under sufficient supervision from the response and carried into matching with minimal loss. The matching vectors are then uploaded to another GRU to form a matching score for the context and the response. The GRU accumulates the pair matching in its hidden states in the chronological order of the utterances in the context. It models relationships and dependencies among the utterances in a matching fashion and has the utterance order supervise the accumulation of pair matching. The match-
ing degree of the context and the response is computed by a logit model with the hidden states of the GRU. SMN extends the powerful “2D” matching paradigm in text pair matching for single-turn conversation to context based matching for multi-turn conversation, and enjoys the advantage that both important information in utterance-response pairs and relationships among utterances are sufficiently preserved and leveraged in matching.
We test our model on the Ubuntu dialogue corpus (Lowe et al., 2015) which is a large scale public English data set for research in multi-turn conversation. The results show that our model can significantly outperform state-of-the-art methods, and improvement to the best baseline model on R10@1 is over 6%. In addition to the Ubuntu corpus, we create a human labeled Chinese data set, namely Douban Conversation Corpus, and test our model on it. Different from the Ubuntu corpus in which data is collected from a specific domain and negative candidates are randomly sampled, conversations in this data come from open domain, and response candidates in this data set are collected from a retrieval engine and labeled by three human judges. On this data, our model improves the best baseline model over 3% on R10@1 and 4% on P@1. As far as we know, Douban Conversation Corpus is the first human labeled data set for multi-turn response selection and could be a good complement to the Ubuntu corpus. We release Douban Conversation Corups and our source code at an anonymous url for blind review. We have uploaded code and data with this paper.
Our contributions in this paper are three-folds: (1) proposal of a new context based matching model for multi-turn response selection in retrieval based chatbots; (2) publication of a large human labeled data set to research communities. (3) empirical verification of the effectiveness of the model on public data sets;
2 Related Work
Recently, building a chatbot with data driven approaches (Ritter et al., 2011; Higashinaka et al., 2014) has drawn a lot of attention. Existing work along this line includes retrieval based methods (Hu et al., 2014; Ji et al., 2014; Wang et al., 2015; Yan et al., 2016; Wu et al., 2016b; Zhou et al., 2016; Wu et al., 2016a) and generation based methods (Shang et al., 2015; Serban et al., 2015; Vinyals and Le, 2015; Li et al., 2015, 2016; Xing
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
.... .... ....
Score
1 2,M M Convolution Pooling
( )L
.... .... ....
1u
1nu
nu
r
Word Embedding GRU1
GRU2
....
1v
1nv
nv
1'nh
Utterance-Response Matching Matching Accumulation
Segment PairsWord Pairs
Matching Prediction
1'h
'nh
Figure 1: Architecture of SMN
et al., 2016; Serban et al., 2016a). Our work belongs to retrieval based methods, and we study context based response selection.
Early studies of retrieval based chatbots focus on response selection for single-turn conversation (Wang et al., 2013; Ji et al., 2014; Wang et al., 2015; Wu et al., 2016b). Recently, researchers begin to pay attention to multi-turn conversation. For example, Lowe et al. (2015) match a response with the literal concatenation of context utterances. Yan et al. (2016) concatenate context utterances with the input message as reformulated queries and perform matching with a deep neural network architecture. Zhou et al. (2016) improve multi-turn response selection with a multi-view model including an utterance view and a word view. Our model is different in that it matches a response with each utterance at first and accumulates matching information instead of sentences by a GRU, thus useful information for matching can be sufficiently retained.
3 Sequential Matching Network
3.1 Problem Formalization
Suppose that we have a data set D = {(yi, si, ri)}Ni=1, where si = {ui,1, . . . , ui,ni} represents a conversation context with {ui,k}nik=1 as utterances. ri is a response candidate and yi ∈ {0, 1} denotes a label. yi = 1 means ri is a proper response for si, otherwise yi = 0. Our goal is to learn a matching model g(·, ·) with D. For any context-response pair (s, r), g(s, r) measures the matching degree between s and r.
3.2 Model Overview
We propose a sequential matching network (SMN) to model g(·, ·). Figure 1 gives the architecture.
SMN first decomposes context-response matching into several utterance-response pair matching and then all pair matching is accumulated as a context based matching through a recurrent neural network. SMN consists of three layers. The first layer matches a response candidate with each utterance in the context on a word level and a segment level, and important matching information from the two levels is distilled by convolution and pooling and encoded in a matching vector. The matching vectors are then fed into the second layer where they are accumulated in the hidden states of a recurrent neural network with GRU following the chronological order of the utterances in the context. The third layer calculates the final matching score with the hidden states of the second layer.
SMN enjoys several advantages over the existing models. First, a response candidate can meet each utterance in the context at the very beginning, thus matching information in every utteranceresponse pair can be sufficiently extracted and carried to the final matching score with minimal loss. Second, information extraction from each utterance is conducted on different levels of granularity and under sufficient supervision from the response, thus semantic structures that are useful to response selection in each utterance can be well identified and extracted. Third, matching and utterance relationships are coupled rather than separately modeled, thus utterance relationships (e.g., order), as a kind of knowledge, can supervise the formation of the matching score.
By taking utterance relationships into account, SMN extends the “2D” matching that has proven effective in text pair matching for single-turn response selection to sequential “2D” matching for context based matching in response selection for multi-turn conversation. In the following sections,
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
we will describe details of the three layers.
3.3 Utterance-Response Matching
Given an utterance u in a context s and a response candidate r, the model looks up an embedding table and represents u and r as U = [eu,1, . . . , eu,nu ] and R = [er,1, . . . , er,nr ] respectively, where eu,i, er,i ∈ Rd are the embeddings of the i-th word of u and r respectively. U ∈ Rd×nu and R ∈ Rd×nr are then used to construct a word-word similarity matrix M1 ∈ Rnu×nr and a sequence-sequence similarity matrix M2 ∈ Rnu×nr which are two input channels of a convolutional neural network (CNN). The CNN distills important matching information from the matrices and encodes the information into a matching vector v.
Specifically, ∀i, j, the (i, j)-th element of M1 is defined by
e1,i,j = e > u,i · er,j . (1)
M1 models the matching between u and r on a word level.
To construct M2, we first employ a GRU to transform U and R to hidden vectors. Suppose that Hu = [hu,1, . . . , hu,nu ] are the hidden vectors of U, then ∀i, hu,i ∈ Rm is defined by
zi = σ(Wzeu,i +Uzhu,i−1) ri = σ(Wreu,i +Urhu,i−1)
h̃u,i = tanh(Wheu,i +Uh(ri hu,i−1)) hu,i = zi h̃u,i + (1− zi) hu,i−1, (2)
where hu,0 = 0, zi and ri are an update gate and a reset gate respectively, σ(·) is a sigmoid function, and Wz, Wh, Wr, Uz, Ur,Uh are parameters. Similarly, we have Hr = [hr,1, . . . , hr,nr ] as the hidden vectors of R. Then, ∀i, j, the (i, j)-th element of M2 is defined by
e2,i,j = h > u,iAhr,j , (3)
where A ∈ Rm×m is a linear transformation. ∀i, GRU models the sequential relationship and the dependency among words up to position i and encodes the text segment until the i-th word to a hidden vector. Therefore, M2 models the matching between u and r on a segment level. M1 and M2 are then processed by a CNN to form v. ∀f = 1, 2, CNN regards Mf as an input channel, and alternates convolution and max-pooling operations. Suppose that z(l,f) =
[ z (l,f) i,j ] I(l,f)×J(l,f) denotes the output of feature maps of type-f on layer-l, where z(0,f) = Mf , ∀f = 1, 2. On the convolution layer, we employ a 2D convolution operation with a window size r (l,f) w × r(l,f)h , and define z (l,f) i,j as
z (l,f) i,j = σ( Fl−1∑ f ′=0 r (l,f) w∑ s=0 r (l,f) h∑ t=0 W (l,f) s,t · z (l−1,f ′) i+s,j+t + b l,k), (4)
where σ(·) is a ReLU, W(l,f) ∈ Rr (l,f) w ×r (l,f) h and bl,k are parameters, and Fl−1 is the number of feature maps on the (l − 1)-th layer. A max pooling operation follows a convolution operation and can be formulated as
z (l,f) i,j = max
p (l,f) w >s≥0 max p (l,f) h >t≥0 zi+s,j+t, (5)
where p(l,f)w and p (l,f) h are the width and the height of the 2D pooling respectively. The output of the final feature maps are concatenated and mapped to a low dimensional space with a linear transformation as the matching vector v ∈ Rq.
From Equation (1), (3), (4), and (5), we can see that by learning word embedding and parameters of GRU from training data, words or segments in an utterance that are useful to recognize the appropriateness of a response may have high similarity with some words or segments in the response and result in high value areas in the similarity matrices. These areas will be transformed and selected by convolution and pooling operations and carry the important information in the utterance to the matching vector. This is how our model identifies important information in context and leverage it in matching under the supervision of the response. We consider multiple channels because we want to capture important matching information on multiple levels of granularity of text.
3.4 Matching Accumulation
Suppose that [v1, . . . , vn] is the output of the first layer (corresponding to n pairs), at the second layer, a GRU takes [v1, . . . , vn] as an input and encodes the matching sequence into its hidden states Hm = [h ′ 1, . . . , h ′ n] ∈ Rq×n with a detailed parameterization similar to Equation (2). This layer has two functions: (1) it models the dependency and the temporal relationship of utterances in the context; (2) it leverages the temporal relationship to supervise the accumulation of the pair matching as a context based matching. Moreover, from
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Equation (2), we can see that the reset gate (i.e., ri) and the update gate (i.e., zi) control how much information from the previous hidden state and the current input flows to the current hidden state, thus important matching vectors (corresponding to important utterances) can be accumulated while noise in the vectors can be filtered out.
3.5 Matching Prediction and Learning
With [h′1, . . . , h ′ n], we define g(s, r) as
g(s, r) = softmax(W2L[h ′ 1, . . . , h ′ n] + b2), (6)
where W2 and b2 are parameters. We consider three parameterizations for L[h′1, . . . , h ′ n]: (1) only the last hidden state is used. Then L[h′1, . . . , h ′ n] = h ′ n. (2) the hidden states
are linearly combined. Then, L[h′1, . . . , h ′ n] =∑n
i=1wih ′ i, where wi ∈ R. (3) we follow (Yang et al., 2016) and employ an attention mechanism to combine the hidden states. Then, L[h′1, . . . , h ′ n] is defined as
ti = tanh(W1,1hui,nu +W1,2h ′ i + b1), αi = exp(t>i ts)∑ i(exp(t > i ts)) ,
L[h′1, . . . , h ′ n] = n∑ i=1 αih ′ i, (7)
where W1,1 ∈ Rq×m,W1,2 ∈ Rq×q and b1 ∈ Rq are parameters. h′i and hui,nu are the i-th matching vector and the final hidden state of the i-th utterance respectively. ts ∈ Rq is a virtual context vector which is randomly initialized and jointly learned in training.
Both (2) and (3) aim to learn weights for {h′1, . . . , h′n} from training data and highlight the effect of important matching vectors in the final matching. The difference is that weights in (2) are static, because the weights are totally determined by the positions of utterances, while weights in (3) are dynamically computed by the matching vectors and utterance vectors. We denote our model with the three parameterizations of L[h′1, . . . , h ′ n] as SMNlast, SMNstatic, and SMNdynamic, and empirically compare them in experiments.
We learn g(·, ·) by minimizing cross entropy withD. Let Θ denote the parameters of SMN, then the objective function L(D,Θ) of learning can be formulated as
− N∑ i=1 [yilog(g(si, ri)) + (1− yi)log(1− g(si, ri))] . (8)
4 Response Candidate Retrieval
In practice of a retrieval based chatbot, to apply the matching approach to response selection, one needs to retrieve a bunch of response candidates from an index beforehand. While candidate retrieval is not the focus of the paper, it is an important step in a real system. In this work, we exploit a heuristic method to obtain response candidates from the index. Given a message un with {u1, . . . , un−1} utterances in its previous turns, we extract top 5 keywords from {u1, . . . , un−1} based on their tf-idf scores1 and expand un with the keywords. Then we send the expanded message to the index and retrieve response candidates using the inline retrieval algorithm of the index. Finally, we use g(s, r) to re-rank the candidates and return the top one as a response to the context.
5 Experiments
We tested our model on a public English data set and a Chinese data set we publish with this paper.
5.1 Ubuntu Corpus
The English data set is the Ubuntu Corpus (Lowe et al., 2015) which contains multi-turn dialogues collected from chat logs of Ubuntu Forum. The data set consists of 1 million context-response pairs for training, 0.5 million pairs for validation, and 0.5 million pairs for test. Positive responses are true responses from human, and negative ones are randomly sampled. The ratio of the positive and the negative is 1:1 in training, and 1:9 in validation and test. We used the copy shared by Xu et al. (2016) 2 in which numbers, urls, and paths are replaced by special placeholders. We followed (Lowe et al., 2015) and employed recall at position k in n candidates (Rn@k) as evaluation metrics.
5.2 Douban Conversation Corpus
Ubuntu Corpus is a domain specific data set, and response candidates are obtained from negative sampling without human judgment. To further verify the efficacy of our model, we created a new data set with open domain conversations, namely Douban Conversation Corpus. Response candidates in the test set of Douban Conversation Corpus are collected following the procedure of a re-
1Tf is word frequency in the context, while idf is calculated using the entire index.
2https://www.dropbox.com/s/ 2fdn26rj6h9bpvl/ubuntudata.zip?dl=0
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
trieval based chatbot and are labeled by human judges. Douban Conversation Corpus simulates the real scenario of a retrieval based chatbot, and we publish it to research communities to facilitate the research of multi-turn response selection.
Specifically, we crawled 1.1 million dyadic dialogues (conversation between two persons) longer than 2 turns from Douban group3 which is a popular social networking service in China. From the data, we randomly sampled 0.5 million dialogues for creating a training set, 25 thousand dialouges for creating a validation set, and 1, 000 dialogues for creating a test set, and made sure that there is no overlap among the three sets. For each dialogue in training and validation, we took the last turn as a positive response for the previous turns as a context and randomly sampled another response from the 1.1 million data as a negative response. There are 1 million context-response pairs in the training set and 50 thousand pairs in the validation set.
To create the test set, we first crawled 15 million post-reply pairs from Sina Weibo4 which is the largest microblogging service in China and indexed the pairs with Lucene5. We took the last turn of each Douban dyadic dialogue in the test set as a message, retrieved 10 response candidates from the index following the method in Section 4, and finally formed a test set with 10, 000 context-response pairs. We recruited three labelers to judge if a candidate is a proper response to the context. A proper response means the response can naturally reply to the message given the whole context. Each pair received three labels and the majority of the labels were taken as the final decision. Table 2 gives the statistics of the three sets. Note that the Fleiss’ kappa (Fleiss, 1971) of the labeling is 0.41, which indicates that the three labelers reached a relatively high agreement.
Besides Rn@ks, we also followed the convention of information retrieval and employed mean average precision (MAP) (Baeza-Yates et al., 1999), mean reciprocal rank (MRR) (Voorhees et al., 1999), and precision at position 1 (P@1) as evaluation metrics. We did not calculate R2@1 because in Douban corpus one context could have more than one correct responses, and we have to randomly sample one for R2@1, which may bring bias to evaluation. When using the labeled set,
3https://www.douban.com/group 4http://weibo.com/ 5https://lucenenet.apache.org/
we removed conversations with all negative responses or all positive responses, as models make no difference on them. There are 6, 670 contextresponse pairs left in the test set.
5.3 Baseline
We considered the following baselines:
Basic models: models in (Lowe et al., 2015) and (Kadlec et al., 2015) including TF-IDF, RNN, CNN, LSTM and BiLSTM.
Multi-view: the model proposed by Zhou et al. (2016) that utilizes a hierarchical recurrent neural network to model utterance relationships.
Deep learning to respond (DL2R): the model proposed by Yan et al. (2016) that reformulates the message with other utterances in the context.
Advanced single-turn matching models: since BiLSTM does not represent the state-ofthe-art matching model, we concatenated the utterances in a context and matched the long text with a response candidate using more powerful models including MV-LSTM (Wan et al., 2016) (2D matching), Match-LSTM (Wang and Jiang, 2015), Attentive-LSTM (Tan et al., 2015) (two attention based models), and Multi-Channel which is described in Section 3.3. Multi-Channel is a simple version of our model without considering utterance relationships. We also appended the top 5 tf-idf words in context to the input message, and computed the score between the expanded message and a response with Multi-Channel, denoted as Multi-Channelexp.
5.4 Parameter Tuning
For baseline models, if their results are available in the existing literatures (e.g., those on the Ubuntu corpus), we just copied the numbers, otherwise we implemented the models following the settings in the literatures. All models were implemented using Theano (Theano Development Team, 2016). Word embeddings were initialized by the results of word2vec (Mikolov et al., 2013) which ran on the training data, and the dimensionality of word vectors is 200. For Multi-Channel and layer one of
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Ubuntu Corpus Douban Conversation Corpus R2@1 R10@1 R10@2 R10@5 MAP MRR P@1 R10@1 R10@2 R10@5
TF-IDF 0.659 0.410 0.545 0.708 0.331 0.359 0.179 0.095 0.172 0.405 RNN 0.768 0.403 0.547 0.819 0.390 0.422 0.208 0.011 0.223 0.589 CNN 0.848 0.549 0.684 0.896 0.417 0.440 0.226 0.012 0.252 0.647 LSTM 0.901 0.638 0.784 0.949 0.485 0.527 0.320 0.187 0.343 0.720 BiLSTM 0.895 0.630 0.780 0.944 0.479 0.514 0.313 0.184 0.330 0.716 Multi-View 0.908 0.662 0.801 0.951 0.505 0.543 0.342 0.202 0.350 0.729 DL2R 0.899 0.626 0.783 0.944 0.488 0.527 0.330 0.193 0.342 0.705 MV-LSTM 0.906 0.653 0.804 0.946 0.498 0.538 0.348 0.202 0.351 0.710 Match-LSTM 0.904 0.653 0.799 0.944 0.500 0.537 0.345 0.202 0.348 0.720 Attentive-LSTM 0.903 0.633 0.789 0.943 0.495 0.523 0.331 0.192 0.328 0.718 Multi-Channel 0.904 0.656 0.809 0.942 0.506 0.543 0.349 0.203 0.351 0.709 Multi-Channelexp 0.714 0.368 0.497 0.745 0.476 0.515 0.317 0.179 0.335 0.691 SMNlast 0.923 0.723 0.842 0.956 0.526 0.571 0.392 0.236 0.387 0.729 SMNstatic 0.927 0.725 0.838 0.962 0.523 0.572 0.387 0.228 0.387 0.734 SMNdynamic 0.926 0.726 0.847 0.961 0.529 0.569 0.395 0.233 0.396 0.724
Table 3: Evaluation results on the two data sets. Numbers in bold mean that the improvement is statistically significant compared with the best baseline.
our model, we set the dimensionality of the hidden states of GRU as 200. We tuned the window size of convolution and pooling in {(2, 2), (3, 3)(4, 4)} and chose (3, 3) finally. The number of feature maps is 8. In layer two, we set the dimensionality of matching vectors and the hidden states of GRU as 50. The parameters were updated by stochastic gradient descent with Adam algorithm (Kingma and Ba, 2014) on a single Tesla K80 GPU. The initial learning rate is 0.001, and the parameters of Adam, β1 and β2 are 0.9 and 0.999 respectively. We employed early-stopping as a regularization strategy. Models were trained in minibatches with a batch size 200, and maximum utterance length is 50. We set the maximum context length (i.e., number of utterances) as 10, because performance of models does not get improved on contexts longer than 10 (details are shown in the supplementary material). We padded zeros if the number of utterances in a context is less than 10, otherwise we kept the last 10 utterances.
5.5 Evaluation Results
Table 3 shows the evaluation results on the two data sets. Our models outperform baselines greatly in terms of all metrics on both data sets, and the improvements are statistically significant (t-test with p-value ≤ 0.01, except R10@5 on Douban Corpus). Even the state-of-the-art singleturn matching models perform much worse than our models. The results demonstrate that one cannot neglects utterance relationships and simply perform multi-turn response selection by concatenating utterances together. Our models achieve significant improvements over Multi-View, which justified our “matching first” strategy. DL2R is
worse than our models, indicating that utterance reformulation with heuristic rules is not a good method to utilize context information. Rn@ks are low on Douban corpus as there are multiple correct candidates for a context (e.g., if there are 3 correct responses, then the maximumR10@1 is 0.33). SMNdynamic is only slightly better than SMNstatic and SMNlast. The reason might be that GRU can select useful signals from the matching sequence and accumulate them in the final state with its gate mechanism, thus the efficacy of attention mechanism is not obvious for the task.
5.6 Further Analysis
Visualization: we visualize the similarity matrices and the gates of GRU in layer two using an example from the Ubuntu corpus to further clarify how our model identifies important information in the context and how it selects important matching vectors with the gate mechanism of GRU as described in Section 3.3 and Section 3.4. The example is {u1: how can unzip many rar ( number for example ) files at once; u2: sure you can do that in bash; u3: okay how? u4: are the files all in the same directory? u5: yes they all are; r: then the command glebihan should extract them all from/to that directory}. It is from the test set and our model successfully ranked the correct response to the top position. Due to space limitation, we only visualized M1, M2 and the update gate (i.e. z) in Figure 2. Other pieces of our model are shown in the supplementary material. We can see that in u1 important words including “unzip”, “rar”, “files” are recognized and carried to matching by “command”, “extract”, and “directory” in r, while u3 is almost useless and thus little infor-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Ubuntu Corpus Douban Conversation Corpus R2@1 R10@1 R10@2 R10@5 MAP MRR P@1 R10@1 R10@2 R10@5
ReplaceM 0.905 0.661 0.799 0.950 0.503 0.541 0.343 0.201 0.364 0.729 ReplaceA 0.918 0.716 0.832 0.954 0.522 0.565 0.376 0.220 0.385 0.727 Only M1 0.919 0.704 0.832 0.955 0.518 0.562 0.370 0.228 0.371 0.737 Only M2 0.921 0.715 0.836 0.956 0.521 0.565 0.382 0.232 0.380 0.734 SMNlast 0.923 0.723 0.842 0.956 0.526 0.571 0.392 0.236 0.387 0.729
Table 4: Evaluation results of model ablation.
th en th e
co m
m an
d gl eb ih an sh ou ld ex tra ct th em a ll fro m /toth at di re ct or y
how can
unzip many
rar ( _number_ for
example )
files at once
0.00
0.15
0.30
0.45
0.60
0.75
0.90
1.05
1.20
1.35
1.50
v a lu
e
(a) M1 of u1 and r
th en th e
co m
m an
d
gl eb
ih an
sh ou
ld ex tr ac
t th em a ll
fro m
/toth at
di re
ct or
y
okay how 0.00 0.15 0.30 0.45 0.60 0.75 0.90 1.05 1.20 1.35 1.50 v a lu e
(b) M1 of u3 and r
0 10 20 30 40
u_1
u_2
u_3
u_4
u_5
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
v a lu
e
(c) Update gate
Figure 2: Model visualization. Darker areas mean larger value.
mation is extracted from it. u1 is crucial to response selection and nearly all information from u1 and r flows to the hidden state of GRU, while other utterances are less informative and the corresponding gates are almost “closed” to keep the information from u1 and r until the final state.
Model ablation: we investigate the effect of different parts of SMN by removing them one by one from SMNlast, shown in Table 4. First, replacing the multi-channel “2D” matching with a neural tensor network (NTN) (Socher et al., 2013) (denoted as ReplaceM ) makes the performance drop dramatically. This is because NTN only matches a pair by an utterance vector and a response vector and loses important information in the pair. Together with the visualization, we can conclude that “2D” matching plays a key role in the “matching first” strategy as it captures the important matching information in each pair with minimal loss. Second, the performance slightly drops when replacing the GRU for matching accumulation with a multi-layer perceptron (denoted as ReplaceA). This indicates that utterance relationships are useful. Finally, we left only one channel in matching and found that M2 is a little more powerful than M1 and we achieve the best results with both of them (except on R10@5 on Douban Corpus).
Context length: we study how our model (SMNlast) performs across the length of contexts. Figure 3 shows the comparison on MAP in different length intervals on the Douban corpus. Our model consistently performs better than the baselines, and when contexts become longer, the gap becomes larger. The results demonstrate that our model can well capture the dependencies, espe-
cially long dependencies, among utterances in contexts. We give the comparisons on other metrics in our supplementary material.
(2,5] (5,10] (10,) context length
40
45
50
55 60 M A P
LSTM MV-LSTM Multi-View SMN
Figure 3: Comparison across context length Retrieval v.s. Generation: we compared SMN with a state-of-the-art response generation model VHERD (Serban et al., 2016b) which was trained using D on the Douban corpus. We conducted a side-by-side human comparison on the top one responses of the two models for each context in the test set. The result is that SMN wins on 238 examples, loses on 207 examples, and is comparable with VHRED on the remaining 555 examples. This indicates that a retrieval based chatbot with SMN can provide a better experience than the state-of-the-art generation model in practice.
6 Conclusion and Future Work
We present a new context based model for multiturn response selection in retrieval-based chatbots. Experiment results on public data sets show that the model can significantly outperform the stateof-the-art methods. Besides, we publish the first human labeled multi-turn response selection data set to research communities. In the future, we are going to study how to model logical consistency of responses and improve candidate retrieval (see supplementary material).
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths:  Relatively clear description of context and structure of proposed approach. Relatively complete description of the math. Comparison to an extensive set of alternative systems.  - Weaknesses:  Weak results summary of  side-by-side human  comparison in Section 5. Some disfluency agrammaticality.  - General Discussion:  The article proposes a principled means of modeling utterance context, consisting of a sequence of previous utterances. Some minor issues:  1. Past turns in Table 1 could be numbered, making the text associated with this table (lines 095-103) less difficult to ingest. Currently, readers need to count turns from the top when identifying references in the authors' description, and may wonder whether  second ,  third , and  last  imply a side-specific or global enumeration.  2. Some reader confusion may be eliminated by explicitly defining what  segment  means in  segment level , as occurring on line 269. Previously, on line 129, this seemingly same thing was referred to as  a sequence-sequence [similarity matrix] . The two terms appear to be used interchangeably, but it is not clear what they actually mean, despite the text in section 3.3. It seems the authors may mean  word subsequence  and  word subsequence to word subsequence , where  sub-  implies  not the whole utterance , but not sure.  3. Currently, the variable symbol  n  appears to be used to enumerate words in an utterance (line 306), as well as utterances in a dialogue (line 389). The authors may choose two different letters for these two different purposes, to avoid confusing readers going through their equations.  4. The statement  This indicates that a retrieval based chatbot with SMN can provide a better experience than the state-of-the-art generation model in practice.  at the end of section 5 appears to be unsupported. The two approaches referred to are deemed comparable in 555 out of 1000 cases, with the baseline better than the proposed method in 238 our of the remaining 445 cases. The authors are encouraged to assess and present the statistical significance of this comparison. If it is weak, their comparison permits to at best claim that their proposed method is no worse (rather than  better ) than the VHRED baseline.  5. The authors may choose to insert into Figure 1 the explicit  first layer ,  second layer  and  third layer  labels they use in the accompanying text.  6.  Their is a pervasive use of  to meet  as in  a response candidate can meet each utterace  on line 280 which is difficult to understand.  7. Spelling:  gated recurrent unites ;  respectively  on line 133 should be removed; punctuation on line 186 and 188 is exchanged;  baseline model over  to  baseline model by ;  one cannot neglects .