Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders
While recent neural encoder-decoder models have shown great promise in modeling open-domain conversations, they often generate dull and generic responses. Unlike past work that has focused on diversifying the output of the decoder at word-level to alleviate this problem, we present a novel framework based on conditional variational autoencoders that captures the discourse-level diversity in the encoder. Our model uses latent variables to learn a distribution over potential conversational intents and generates diverse responses using only greedy decoders. We have further developed a novel variant that is integrated with linguistic prior knowledge for better performance. Finally, the training procedure is improved by introducing a bag-of-word loss. Our proposed models have been validated to generate significantly more diverse responses than baseline approaches and exhibit competence in discourse-level decision-making.
1 Introduction
The dialog manager is one of the key components of dialog systems, which is responsible for modeling the decision-making process. Specifically, it typically takes a new utterance and the dialog context as input, and generates discourse-level decisions (Bohus and Rudnicky, 2003; Williams and Young, 2007). Advanced dialog managers usually have a list of potential actions that enable them to have diverse behavior during a conversation, e.g. different strategies to recover from non-understanding (Yu et al., 2016). However, the conventional approach of designing a dialog manager (Williams and Young, 2007) does not
scale well to open-domain conversation models because of the vast quantity of possible decisions. Thus, there has been a growing interest in applying encoder-decoder models (Sutskever et al., 2014) for modeling open-domain conversation (Vinyals and Le, 2015; Serban et al., 2016a). The basic approach treats a conversation as a transduction task, in which the dialog history is the source sequence and the next response is the target sequence. The model is then trained end-to-end on large conversation corpora using the maximum-likelihood estimation (MLE) objective without the need for manual crafting.
However recent research has found that encoder-decoder models tend to generate generic and dull responses (e.g., I don’t know), rather than meaningful and specific answers (Li et al., 2015; Serban et al., 2016b). There have been many attempts to explain and solve this limitation, and they can be broadly divided into two categories (see Section 2 for details): (1) the first category argues that the dialog history is only one of the factors that decide the next response. Other features should be extracted and provided to the models as conditionals in order to generate more specific responses (Xing et al., 2016; Li et al., 2016a); (2) the second category aims to improve the encoder-decoder model itself, including decoding with beam search and its variations (Wiseman and Rush, 2016), encouraging responses that have long-term payoff (Li et al., 2016b), etc.
Building upon the past work in dialog managers and encoder-decoder models, the key idea of this paper is to model dialogs as a one-to-many problem at the discourse level. Previous studies indicate that there are many factors in open-domain dialogs that decide the next response, and it is nontrivial to extract all of them. Intuitively, given a similar dialog history (and other observed inputs), there may exist many valid responses (at the
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
discourse-level), each corresponding to a certain configuration of the latent variables that are not presented in the input. To uncover the potential responses, we strive to model a probabilistic distribution over the distributed utterance embeddings of the potential responses using a latent variable (Figure 1). This allows us to generate diverse responses by drawing samples from the learned distribution and reconstruct their words via a decoder neural network.
Figure 1: Given A’s question, there exists many valid responses from B for different assumptions of the latent variables, e.g., B’s hobby.
Specifically, our contributions are three-fold: 1. We present a novel neural dialog model adapted from conditional variational autoencoders (CVAE) (Yan et al., 2015; Sohn et al., 2015), which introduces a latent variable that can capture discourse-level variations as described above 2. We propose Knowledge-Guided CVAE (kgCVAE), which enables easy integration of expert knowledge and results in performance improvement and model interpretability. 3. We develop a training method in addressing the difficulty of optimizing CVAE for natural language generation (Bowman et al., 2015). We evaluate our models on human-human conversation data and yield promising results in: (a) generating appropriate and discourse-level diverse responses, and (b) showing that the proposed training method is more effective than the previous techniques.
2 Related Work
Our work is related to both recent advancement in encoder-decoder dialog models and generative models based on CVAE.
2.1 Encoder-decoder Dialog Models
Since the emergence of the neural dialog model, the problem of output diversity has received much attention in the research community. Ideal output responses should be both coherent and diverse. However, most models end up with generic and dull responses. To tackle this problem, one line of research has focused on augmenting the input of encoder-decoder models with richer context information, in order to generate more spe-
cific responses. Li et al., (2016a) captured speakers’ characteristics by encoding background information and speaking style into the distributed embeddings, which are used to re-rank the generated response from an encoder-decoder model. Xing et al., (2016) maintain topic encoding based on Latent Dirichlet Allocation (LDA) (Blei et al., 2003) of the conversation to encourage the model to output more topic coherent responses.
On the other hand, many attempts have also been made to improve the architecture of encoderdecoder models. Li et al,. (2015) proposed to optimize the standard encoder-decoder by maximizing the mutual information between input and output, which in turn reduces generic responses. This approach penalized unconditionally high frequency responses, and favored responses that have high conditional probability given the input. Wiseman and Rush (2016) focused on improving the decoder network by alleviating the biases between training and testing. They introduced a searchbased loss that directly optimizes the networks for beam search decoding. The resulting model achieves better performance on word ordering, parsing and machine translation. Besides improving beam search, Li et al., (2016b) pointed out that the MLE objective of an encoder-decoder model is unable to approximate the real-world goal of the conversation. Thus, they initialized a encoderdecoder model with MLE objective and leveraged reinforcement learning to fine tune the model by optimizing three heuristic rewards functions: informativity, coherence, and ease of answering.
2.2 Conditional Variational Autoencoder
The variational autoencoder (VAE) (Kingma and Welling, 2013; Rezende et al., 2014) is one of the most popular frameworks for image generation. The basic idea of VAE is to encode the input x into a probability distribution z instead of a point encoding in the autoencoder. Then VAE applies a decoder network to reconstruct the original input using samples from z. To generate images, VAE first obtains a sample of z from the prior distribution, e.g. N (0, I), and then produces an image via the decoder network. A more advanced model, the conditional VAE (CVAE), is a recent modification of VAE to generate diverse images conditioned on certain attributes, e.g. generating different human faces given skin color (Yan et al., 2015; Sohn et al., 2015). Inspired by CVAE, we view the dialog contexts as the conditional attributes and adapt CVAE
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
to generate diverse responses instead of images. Although VAE/CVAE has achieved impressive results in image generation, adapting this to natural language generators is non-trivial. Bowman et al., (2015) have used VAE with Long-Short Term Memory (LSTM)-based recognition and decoder networks to generate sentences from a latent Gaussian variable. They showed that their model is able to generate diverse sentences with even a greedy LSTM decoder. They also reported the difficulty of training because the LSTM decoder tends to ignore the latent variable. We refer to this issue as the vanishing latent variable problem. Serban et al., (2016b) have applied a latent variable hierarchical encoder-decoder dialog model to introduce utterance-level variations and facilitate longer responses. To improve upon the past models, we firstly introduce a novel mechanism to leverage linguistic knowledge in training end-to-end neural dialog models, and we also propose a novel training technique that mitigates the vanishing latent variable problem.
3 Proposed Models
Figure 2: Graphical models of CVAE (a) and kgCVAE (b)
3.1 Conditional Variational Autoencoder (CVAE) for Dialog Generation
Each dyadic conversation can be represented via three random variables: the dialog context c (context window size k − 1), the response utterance x (the kth utterance) and a latent variable z, which is used to capture the latent distribution over the valid responses. Further, c is composed of the dialog history: the preceding k-1 utterances; conversational floor (1 if the utterance is from the same speaker of x, otherwise 0) and meta features m (e.g. the topic). We then define the conditional distribution p(x, z|c) = p(x|z, c)p(z|c) and our goal is to use deep neural networks (parametrized by θ) to approximate p(z|c) and p(x|z, c). We refer to pθ(z|c) as the prior network and pθ(x, |z, c) as the
response decoder. Then the generative process of x is (Figure 2 (a)):
1. Sample a latent variable z from the prior network pθ(z|c).
2. Generate x through the response decoder pθ(x|z, c).
CVAE is trained to maximize the conditional log likelihood of x given c, which involves an intractable marginalization over the latent variable z. As proposed in (Sohn et al., 2015; Yan et al., 2015), CVAE can be efficiently trained with the Stochastic Gradient Variational Bayes (SGVB) framework (Kingma and Welling, 2013) by maximizing the variational lower bound of the conditional log likelihood. We assume the z follows multivariate Gaussian distribution with a diagonal covariance matrix and introduce a recognition network qφ(z|x, c) to approximate the true posterior distribution p(z|x, c). Sohn and et al,. (2015) have shown that the variational lower bound can be written as:
L(θ, φ;x, c) = −KL(qφ(z|x, c)‖pθ(z|c)) +Eqφ(z|c,x)[log pθ(x|z, c)] (1) ≤ log p(x|c)
Figure 3 demonstrates an overview of our model. The utterance encoder is a bidirectional recurrent neural network (BRNN) (Schuster and Paliwal, 1997) with a gated recurrent unit (GRU) (Chung et al., 2014) to encode each utterance into fixedsize vectors by concatenating the last hidden states of the forward and backward RNN ui = [~hi, ~hi]. x is simply uk. The context encoder is a 1-layer GRU network that encodes the preceding k-1 utterances by taking u1:k−1 and the corresponding conversation floor as inputs. The last hidden state hc of the context encoder is concatenated with meta features and c = [hc,m]. Since we assume z follows isotropic Gaussian distribution, the recognition network qφ(z|x, c) ∼ N (µ, σ2I) and the prior network pθ(z|c) ∼ N (µ′, σ′2I), and then we have: [
µ log(σ2)
] =Wr [ x c ] + br (2)[
µ′
log(σ′2)
] = MLPp(c) (3)
We then use the reparametrization trick (Kingma and Welling, 2013) to obtain samples of z either
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Figure 3: The neural network architectures for the baseline and the proposed CVAE/kgCVAE models.⊕ denotes the concatenation of the input vectors. The dashed blue connections only appear in kgCVAE.
from N (z;µ, σ2I) predicted by the recognition network (training) or N (z;µ′, σ′2I) predicted by the prior network (testing). Finally, the response decoder is a 1-layer GRU network with initial state s0 =Wi[z, c]+bi. The response decoder then predicts the words in x sequentially.
3.2 Knowledge-Guided CVAE (kgCVAE)
In practice, training CVAE is a challenging optimization problem and often requires large amount of data. On the other hand, past research in spoken dialog systems and discourse analysis has suggested that many linguistic cues capture crucial features in representing natural conversation. For example, dialog acts (Poesio and Traum, 1998) have been widely used in the dialog managers (Litman and Allen, 1987; Raux et al., 2005; Zhao and Eskenazi, 2016) to represent the propositional function of the system. Therefore, we conjecture that it will be beneficial for the model to learn meaningful latent z if it is provided with explicitly extracted discourse features during the training.
In order to incorporate the linguistic features into the basic CVAE model, we first denote the set of linguistic features as y. Then we assume that the generation of x depends on c, z and y. y relies on z and c as shown in Figure 2. Specifically, during training the initial state of the response decoder is s0 = Wi[z, c, y] + bi and the input at every step is [et, y] where et is the word embedding of tth word in x. In addition, there is an MLP to predict y′ = MLPy(z, c) based on z and c. In the testing stage, the predicted y′ is used by the response decoder instead of the oracle decoders. We denote the modified model as knowledge-guided
CVAE (kgCVAE) and developers can add desired discourse features that they wish the latent variable z to capture. KgCVAE model is trained by maximizing:
L(θ, φ;x, c, y) = −KL(qφ(z|x, c, y)‖Pθ(z|c)) +Eqφ(z|c,x,y)[log p(x|z, c, y)] +Eqφ(z|c,x,y)[log p(y|z, c)] (4)
Since now the reconstruction of y is a part of the loss function, kgCVAE can more efficiently encode y-related information into z than discovering it only based on the surface-level x and c. Another advantage of kgCVAE is that it can output a highlevel label (e.g. dialog act) along with the wordlevel responses, which allows easier interpretation of the model’s outputs.
3.3 Optimization Challenges
A straightforward VAE with RNN decoder fails to encode meaningful information in z due to the vanishing latent variable problem (Bowman et al., 2015). Bowman et al., (2015) proposed two solutions: (1) KL annealing: gradually increasing the weight of the KL term from 0 to 1 during training; (2) word drop decoding: setting a certain percentage of the target words to 0. We found that CVAE suffers from the same issue when the decoder is an RNN. Also we did not consider word drop decoding because Bowman et al,. (2015) have shown that it may hurt the performance when the drop rate is too high.
As a result, we propose a simple yet novel technique to tackle the vanishing latent variable problem: bag-of-word loss. The idea is to introduce
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
an auxiliary loss that requires the decoder network to predict the bag-of-words in the response x as shown in Figure 3(b). We decompose x into two variables: xo with word order and xbow without order, and assume that xo and xbow are conditionally independent given z and c: p(x, z|c) = p(xo|z, c)p(xbow|z, c)p(z|c). Due to the conditional independence assumption, the latent variable is forced to capture global information about the target response. Let f = MLPb(z, x) ∈ RV where V is vocabulary size, and we have:
log p(xbow|z, c) = log |x|∏ t=1 efxt∑V j e fj (5)
where |x| is the length of x and xt is the word index of tth word in x. The modified variational lower bound for CVAE with bag-of-word loss is (see Appendix A for kgCVAE):
L′(θ, φ;x, c) = L(θ, φ;x, c) +Eqφ(z|c,x,y)[log p(xbow|z, c)] (6)
We will show that the bag-of-word loss in Equation 6 is very effective against the vanishing latent variable and it is also complementary to the KL annealing technique.
4 Experiment Setup
4.1 Dataset
We chose the Switchboard (SW) 1 Release 2 Corpus (Godfrey and Holliman, 1997) to evaluate the proposed models. SW has 2400 two-sided telephone conversations with manually transcribed speech and alignment. In the beginning of the call, a computer operator gave the callers recorded prompts that define the desired topic of discussion. There are 70 available topics. We randomly split the data into 2316/60/62 dialogs for train/validate/test. The pre-processing includes (1) tokenize using the NLTK tokenizer (Bird et al., 2009); (2) remove non-verbal symbols and repeated words due to false starts; (3) keep the top 10K frequent word types as the vocabulary. The final data have 207, 833/5, 225/5, 481 (c, x) pairs for train/validate/test. Furthermore, a subset of SW was manually labeled with dialog acts (Stolcke et al., 2000). We extracted dialog act labels based on the dialog act recognizer proposed in (Ribeiro et al., 2015). The features include the uni-gram and bi-gram of the utterance, and the contextual features of the last 3 utterances. We trained a Support Vector Machine
(SVM) (Suykens and Vandewalle, 1999) with linear kernel on the subset of SW with human annotations. There are 42 types of dialog acts and the SVM achieved 77.3% accuracy on held-out data. Then the rest of SW data are labelled with dialog acts using the trained SVM dialog act recognizer.
4.2 Training
We trained with the following hyperparameters (according to the loss on the validate dataset): word embedding has size 200 and is shared across everywhere. We initialize the word embedding from Glove embedding pre-trained on Twitter (Pennington et al., 2014). The utterance encoder has a hidden size of 300 for each direction. The context encoder has a hidden size of 600 and the response decoder has a hidden size of 400. The prior network and the MLP for predicting y both have 1 hidden layer of size 400 and tanh non-linearity. The latent variable z has a size of 200. The context window k is 10. All the initial weights are sampled from a uniform distribution [-0.08, 0.08]. The mini-batch size is 30. The models are trained end-to-end using the Adam optimizer (Kingma and Ba, 2014) with a learning rate of 0.001 and gradient clipping at 5. We selected the best models based on the variational lower bound on the validate data. Finally, we use the BOW loss along with KL annealing of 10,000 batches to achieve the best performance. Section 5.4 gives a detailed argument for the importance of the BOW loss.
5 Results
5.1 Experiments Setup
We compared three neural dialog models: a strong baseline model, CVAE, and kgCVAE. The baseline model is an encoder-decoder neural dialog model without latent variables similar to (Serban et al., 2016a). The baseline model’s encoder uses the same context encoder to encode the dialog history and the meta features as shown in Figure 3. The encoded context c is directly fed into the decoder networks as the initial state. The hyperparameters of the baseline are the same as the ones reported in Section 4.2 and the baseline is trained to minimize the standard cross entropy loss of the decoder RNN model without any auxiliary loss.
Also, to compare the diversity introduced by the stochasticity in the proposed latent variable versus the softmax of RNN at each decoding step, we generate N responses from the baseline by sam-
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
pling from the softmax. For CVAE/kgCVAE, we sample N times from the latent z and only use greedy decoders so that the randomness comes entirely from the latent variable z.
5.2 Quantitative Analysis
Automatically evaluating an open-domain generative dialog model is an open research challenge (Liu et al., 2016). Following our one-tomany hypothesis, we propose the following metrics. We assume that for a given dialog context c, there existMc reference responses rj , j ∈ [1,Mc]. Meanwhile a model can generateN hypothesis responses hi, i ∈ [1, N ]. The generalized responselevel precision/recall for a given dialog context is:
precision(c) =
∑N i=1maxj∈[1,Mc]d(rj , hi)
N
recall(c) =
∑Mc j=1maxi∈[1,N ]d(rj , hi))
Mc
where d(rj , hi) is a distance function which lies between 0 to 1 and measures the similarities between rj and hi. The final score is averaged over the entire test dataset and we report the performance with 3 types of distance functions in order to evaluate the systems from various linguistic points of view:
1. Smoothed Sentence-level BLEU (Chen and Cherry, 2014): BLEU is a popular metric that measures the geometric mean of modified ngram precision with a length penalty (Papineni et al., 2002; Li et al., 2015). We use BLEU-1 to 4 as our lexical similarity metric.
2. Cosine Distance of Bag-of-word Embedding: a simple method to obtain sentence embeddings is to take the average or extrema of all the word embeddings in the sentences (Forgues et al., 2014; Adi et al., 2016). The d(rj , hi) is the cosine distance of the two embedding vectors. We used Glove embedding described in Section 4 and denote the average method as A-bow and extrema method as E-bow.
3. Dialog Act Match: to measure the similarity at the discourse level, the same dialogact tagger from 4.1 is applied to label all the generated responses of each model. We set d(rj , hi) = 1 if rj and hi have the same dialog acts, otherwise d(rj , hi) = 0.
One challenge of using the above metrics is that there is only one, rather than multiple reference responses/contexts. This impacts reliability of our measures. Inspired by (Sordoni et al., 2015), we utilized information retrieval techniques (see Appendix A) to gather 10 extra candidate reference responses/context from other conversations with the same topics. Then the 10 candidate references are filtered by two experts, which serve as the ground truth to train the reference response classifier. The result is 6.69 references in average per context. The average number of distinct reference dialog acts is 4.2. Table 1 shows the results.
Both CVAE and kgCVAE outperform the baseline in terms of recall in all the metrics. This confirms our hypothesis that generating responses with discourse-level diversity can lead to a more comprehensive coverage of the potential responses than promoting only word-level diversity. As for precision, we observed that the baseline has higher or similar scores than CVAE in all metrics, which is expected since the baseline tends to generate the mostly likely and safe responses repeatedly in the N hypotheses. However, kgCVAE is able to achieve the highest precision and recall in the 4 metrics at the same time (BLEU1/3/4, E-BOW). One reason for kgCVAE’s good performance is that the predicted dialog act label in kgCVAE can regularize the generation process of its RNN decoder by forcing it to generate more coherent
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
and precise words. We further analyze the precision/recall of BLEU-4 by looking at the average score versus the number of distinct reference dialog acts. A low number of distinct dialog acts represents the situation where the dialog context has a strong constraint on the range of the next response (low entropy), while a high number indicates the opposite (high-entropy). Figure 4 shows that CVAE/kgCVAE achieves significantly higher recall than the baseline in higher entropy contexts. Also it shows that CVAE suffers from lower precision, especially in low entropy contexts. Finally, kgCVAE gets higher precision than both the baseline and CVAE in the full spectrum of context entropy.
Figure 4: BLEU-4 precision/recall vs. the number of distinct reference dialog acts.
5.3 Qualitative Analysis
Table 2 shows the outputs generated from the baseline and kgCVAE. In example 1, caller A begins with an open-ended question. The kgCVAE model generated highly diverse answers that cover multiple plausible dialog acts. Further, we notice that the generated text exhibits similar dialog acts compared to the ones predicted separately by the model, implying the consistency of natural language generation based on y. On the contrary, the responses from the baseline model are limited to local n-gram variations and share a similar prefix, i.e. ”I’m”. Example 2 is a situation where caller A is telling B stories. The ground truth response is a back-channel and the range of valid answers is more constrained than example 1 since B is playing the role of a listener. The baseline successfully predicts ”uh-huh”. The kgCVAE model is also able to generate various ways of back-channeling. This implies that the latent z is able to capture context-sensitive variations, i.e. in low-entropy dialog contexts modeling lexical diversity while in high-entropy ones modeling discourse-level diversity. Moreover, kgCVAE is occasionally able to generate more sophisticated grounding (sample 4) beyond a simple back-channel, which is also an acceptable response given the dialog context.
In addition, past work (Kingma and Welling, 2013) has shown that the recognition network is able to learn to cluster high-dimension data, so we conjecture that posterior z outputted from the recognition network should cluster the responses into meaningful groups. Figure 5 visualizes the posterior z of responses in the test dataset in 2D space using t-SNE (Maaten and Hinton, 2008). We found that the learned latent space is highly correlated with the dialog act and length of responses, which confirms our assumption.
Figure 5: t-SNE visualization of the posterior z for test responses with top 8 frequent dialog acts. The size of circle represents the response length.
5.4 Results for Bag-of-Word Loss
Finally, we evaluate the effectiveness of bag-ofword (BOW) loss for training VAE/CVAE with the RNN decoder. To compare with past work (Bowman et al., 2015), we conducted the same language modelling (LM) task on Penn Treebank using VAE. The network architecture is same except we use GRU instead of LSTM. We compared four different training setups: (1) standard VAE without any heuristics; (2) VAE with KL annealing (KLA); (3) VAE with BOW loss; (4) VAE with both BOW loss and KLA. Intuitively, a well trained model should lead to a low reconstruction loss and small but non-trivial KL cost. For all models with KLA, the KL weight increases linearly from 0 to 1 in the first 5000 batches.
Table 3 shows the reconstruction perplexity and the KL cost on the test dataset. The standard VAE fails to learn a meaningful latent variable by having a KL cost close to 0 and a reconstruction perplexity similar to a small LSTM LM (Zaremba et al., 2014). KLA helps to improve the recon-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Example 1-Topic: Recycling Context: A: are they doing a lot of recycling out in Georgia? Target-B (statement): well at my workplace we have palaces for aluminium cans and we have a separate trash can for recyclable paper Baseline+Sampling kgCVAE+Greedy 1. well I’m a graduate student and have two kids 1. (non-understand) pardon 2. well I was in last year and so we’ve had lots of recycling 2. (statement) oh you’re not going to have a curbside pick up here 3. I’m not sure 3. (statement) okay I am sure about a recycling center 4. well I don’t know I just moved here in new york 4. (yes-answer) yeah so
Example 2-Topic: Child Care Context: A: you know a private home to take their children to when they’re young until they hit the preschool age and they Target-B (backchannel): uh-huh Baseline+Sampling kgCVAE+Greedy 1. um - hum 1. (backchannel) uh-huh 2. yeah 2. (turn-exit) um-hum 3. um - hum 3. (backchannel) yeah 4. uh-huh 4. (statement) oh yeah I think that’s part of the problem
Table 2: Generated responses from the baselines and kgCVAE in two examples. KgCVAE also provides the predicted dialog act for each response. The context only shows the last utterance due to space limit (the actual context window size is 10).
struction loss, but it requires early stopping since the models will fall back to the standard VAE after the KL weight becomes 1. At last, the models with BOW loss achieved significantly lower perplexity and larger KL cost.
Model Perplexity KL cost Standard 122.0 0.05 KLA 111.5 2.02 BOW 97.72 7.41 BOW+KLA 73.04 15.94
Table 3: The reconstruction perplexity and KL terms on Penn Treebank test set.
Figure 6 visualizes the evolution of the KL cost. We can see that for the standard model, the KL cost crashes to 0 at the beginning of training and never recovers. On the contrary, the model with only KLA learns to encode substantial information in latent z when the KL cost weight is small. However, after the KL weight is increased to 1 (after 5000 batch), the model once again decides to ignore the latent z and falls back to the naive implementation. The model with BOW loss, however, consistently converges to a non-trivial KL cost even without KLA, which confirms the importance of BOW loss for training latent variable models with the RNN decoder. Last but not least, our experiments showed that the conclusions drawn from LM using VAE also apply to training CVAE/kgCVAE, so we used BOW loss together with KLA for all previous experiments.
6 Conclusion and Future Work
In conclusion, we identified the one-to-many nature of open-domain conversation and proposed two novel models that show superior performance in generating diverse and appropriate responses at the discourse level. While the current paper addresses diversifying responses in respect to dialogue acts, this work is part of a larger research direction that targets leveraging both past linguistic findings and the learning power of deep neural networks to learn better representation of the latent factors in dialog. In turn, the output of this novel neural dialog model will be easier to explain and control by humans. In addition to dialog acts, we plan to apply our kgCVAE model to capture other different linguistic phenomena including sentiment, named entities,etc. Last but not least, the recognition network in our model will serve as the foundation for designing a datadriven dialog manager, which automatically discovers useful high-level intents. All of the above suggest a promising research direction.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
A Supplemental Material
Variational Lower Bound for kgCVAE We assume that even with the presence of linguistic feature y regarding x, the prediction of xbow still only depends on the z and c. Therefore, we have:
L(θ, φ;x, c, y) = −KL(qφ(z|x, c, y)‖Pθ(z|c)) +Eqφ(z|c,x,y)[log p(x|z, c, y)] +Eqφ(z|c,x,y)[log p(y|z, c)] +Eqφ(z|c,x,y)[log p(xbow|z, c)]
(7)
Collection of Multiple Reference Responses We collected multiple reference responses for each dialog context in the test set by information retrieval techniques combining with traditional a machine learning method. First, we encode the dialog history using Term Frequency-Inverse Document Frequency (TFIDF) (Salton and Buckley, 1988) weighted bag-of-words into vector representation h. Then we denote the topic of the conversation as t and denote f as the conversation floor, i.e. if the speakers of the last utterance in the dialog history and response utterance are the same f = 1 otherwise f = 0. Then we computed the similarity d(ci, cj) between two dialog contexts using:
d(ci, cj) = 1(ti = tj)1(ti = tj) hi · hj ||hi||||hj || (8)
Unlike past work (Sordoni et al., 2015), this similarity function only cares about the distance in the context and imposes no constraints on the response, therefore is suitbale for finding diverse responses regarding to the same dialog context. Secondly, for each dialog context in the test set, we retrieved the 10 nearest neighbors from the training set and treated the responses from the training set as candidate reference responses. Thirdly, we further sampled 240 context-responses pairs from 5481 pairs in the total test set and post-processed the selected candidate responses by two human computational linguistic experts who were told to give a binary label for each candidate response about whether the response is appropriate regarding its dialog context. The filtered lists then served as the ground truth to train our reference response classifier. For the next step, we extracted bigrams, part-of-speech bigrams and word part-of-speech
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
pairs from both dialogue contexts and candidate reference responses with rare threshold for feature extraction being set to 20. Then L2-regularized logistic regression with 10-fold cross validation was applied as the machine learning algorithm. Cross validation accuracy on the human-labelled data was 71%. Finally, we automatically annotated the rest of test set with this trained classifier and the resulting data were used for model evaluation.

Review, ACL 2017, paper 256:  This paper extends the line of work which models generation in dialogue as a sequence to sequence generation problem, where the past N-1 utterances (the  dialogue context ) are encoded into a context vector (plus potential other, hand-crafted features), which is then decoded into a response: the Nth turn in the dialogue. As it stands, such models tend to suffer from lack of diversity, specificity and local coherence in the kinds of response they tend to produce when trained over large dialogue datasets containing many topics (e.g. Cornell, Opensubtitles, Ubuntu, etc.). Rather than attempting to produce diverse responses using the decoder, e.g. through word-by-word beam search (which has been shown not to work very well, even lose crucial information about grammar and valid sequences), or via a different objective function (such as in Li et. al.  work) the authors introduce a latent variable, z, over which a probability distribution is induced as part of the network. At prediction time, after encoding utterances 1 to k, a context z is sampled, and the decoder is greedily used to generate a response from this. The evaluation shows small improvements in BLEU scores over a vanilla seq2seq model that does not involve learning a probability distribution over contexts and sampling from this.  The paper is certainly impressive from a technical point of view, i.e. in the application of deep learning methods, specifically conditioned variational auto encoders, to the problem of response generation, and its attendant difficulties in training such models. Their use of Information-Retrieval techniques to get more than one reference response is also interesting.   I have some conceptual comments on the introduction and the motivations behind the work, some on the model architecture, and the evaluation which I write below in turn:  Comments on the introduction and motivations u2026.   The authors seem not fully aware of the long history of this field, and its various facets, whether from a theoretical perspective, or from an applied one.  1.  [the dialogue manager] typically takes a new utterance and the dialogue context as input, and generates discourse level decisions. d           This is not accurate. Traditionally at least, the job of the dialogue manager is to select actions (dialogue acts) in a particular dialogue context. The                    action chosen is then passed to a separate generation module for realisation. Dialogue management is usually done in the context of task-based systems which are goal driven. The dialogue manager is to choose actions which are optimal in some sense, e.g. reach a goal (e.g. book a restaurant) in as few steps as possible. See publications from Lemon & Pietquin, 2012, Rieser, Keizer and colleagues, and various publications from Steve Young, Milica Gasic and colleagues for an overview of the large literature on Reinforcement Learning and MDP models for task-based dialogue systems.  2. The authors need to make a clear distinction between task-based, goal-oriented dialogue, and chatbots social bots, the latter being usually no more than a language model, albeit a sophisticated one (though see Wen et. al. 2016). What is required from these two types of system is usually distinct. Whereas the former is required to complete a task, the latter is, perhaps only required to keep the user engaged. Indeed the data-driven methods that have been used to build such systems are usually very different.  3. The authors refer to  open-domain  conversation. I would suggest that there is no such thing as open-domain conversation - conversation is always in the context of some activity and for doing achieving something specific in the world. And it is this overarching goal, the overarching activity, this overarching genre, which determines the outward shape of dialogues and determines what sorts of dialogue structure are coherent. Coherence itself is activity context-specific. Indeed a human is not capable of open-domain dialogue: if they are faced with a conversational topic or genre that they have never participated in, they would embarrass themselves with utterances that would look incoherent and out of place to others already familiar with it. (think of a random person on the street trying to follow the conversations at some coffee break at ACL). This is the fundamental problem I see with systems that attempt to use data from an EXTREMELY DIVERSE, open-ended set of conversational genres (e.g. movie subtitles) in order to train one model, mushing everything together so that what emerges at the other end is just very good grammatical structure. Or very generic responses.   Comments on the model architecture:  Rather than generate from a single encoded context, the authors induce a distribution over possible contexts, sample from this, and generate greedily with the decoder. It seems to me that this general model is counter intuitive, and goes against evidence from the Linguistic Psycholinguistic literature on dialogue: this literature shows that people tend to resolve potential problems in understanding and acceptance very locally - i.e. make sure they agree on what the context of the conversation is - and only then move on with the rest of the conversation, so that at any given point, there is little uncertainty about the current context of the conversation. The massive diversity one sees results from the diversity in what the conversation is actually trying to achieve (see above), diversity in topics and contexts etc, so that in a given, fixed context, there is a multitude of possible next actions, all coherent, but leading the conversation down a different path.  It therefore seems strange to me at least to shift the burden of explaining diversity and coherence in follow-up actions to that of the linguistic verbal surface contexts in which they are uttered, though of course, uncertainty here can also arise as a result of mismatches in vocabulary, grammars, concepts, people  backgrounds etc. But this probably wouldn t explain much of the variation in follow-up response.   In fact, at least as far as task-based Dialogue systems are concerned, the challenge is to capture synonymy of contexts, i.e. dialogues that are distinct on the surface, but lead to the same or similar context, either in virtue of interactional and syntactic equivalence relations, or synonymy relations that might hold in a particular domain between words or sequences of words (e.g.  what is your destination? d =  where would you like to go? d in a flight booking domain). See e.g. Bordes & Weston, 2016; and Kalatzis, Eshghi & Lemon, 2016 - the latter use a grammar to cluster semantically similar dialogues.  Comments on the evaluation:  The authors seek to show that their model can generate more coherent, and more diverse responses. The evaluation method, though very interesting, seems to address coherence but not diversity, despite what they say in section 5.2:  The precision and recall metrics measure distance between ground truth utterances and the ones the model generates, but not that between the generated utterances themselves (unless I m misunderstanding the evaluation method). See e.g. Li et al. who measure diversity by counting the number distinct n-grams in the generated responses.  Furthermore, I m not sure that the increase in BLEU scores are meaningful: they are very small. In the qualitative assessment of the generated responses, one certainly sees more diversity, and more contentful utterances in the examples provided. But I can t see how frequent such cases in fact are.  Also, it would have made for a stronger, more meaningful paper if the authors had compared their results with other work, (e.g. Li et. al) that use very different methods to promote diversity (e.g. by using a different objective function). The authors in fact do not mention this, or characterise it properly, despite actually referring to Li et. al. 2015.


Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search
We present Grid Beam Search (GBS), an algorithm which extends beam search to allow the inclusion of pre-specified lexical constraints. The algorithm can be used with any model that generates a sequence ŷ = {y0 . . . yT }, by maximizing p(y|x) = ∏ t p(yt|x; {y0 . . . yt−1}). Lexical constraints take the form of phrases or words that must be present in the output sequence. This is a very general way to incorporate additional knowledge into a model’s output without requiring any modification of the model parameters or training data. We demonstrate the feasibility and flexibility of Lexically Constrained Decoding by conducting experiments on Neural Interactive-Predictive Translation, as well as Domain Adaptation for Neural Machine Translation. Experiments show that GBS can provide large improvements in translation quality in interactive scenarios, and that, even without any user input, GBS can be used to achieve significant gains in performance in domain adaptation scenarios.
We present Grid Beam Search (GBS), an algorithm which extends beam search to allow the inclusion of pre-specified lexical constraints. The algorithm can be used with any model that generates a sequence ŷ = {y0 . . . yT }, by maximizing p(y|x) =
∏ t p(yt|x; {y0 . . . yt−1}). Lex-
ical constraints take the form of phrases or words that must be present in the output sequence. This is a very general way to incorporate additional knowledge into a model’s output without requiring any modification of the model parameters or training data. We demonstrate the feasibility and flexibility of Lexically Constrained Decoding by conducting experiments on Neural Interactive-Predictive Translation, as well as Domain Adaptation for Neural Machine Translation. Experiments show that GBS can provide large improvements in translation quality in interactive scenarios, and that, even without any user input, GBS can be used to achieve significant gains in performance in domain adaptation scenarios.
1 Introduction
The output of many natural language processing models is a sequence of text. Examples include automatic summarization (Rush et al., 2015), machine translation (Koehn, 2010; Bahdanau et al., 2014), caption generation (Xu et al., 2015), and dialog generation (Serban et al., 2016), among others.
In many real-world scenarios, additional information that could inform the search for the optimal output sequence may be available at inference
time. Humans can provide corrections after viewing a system’s initial output, or separate classification models may be able to predict parts of the output with high confidence. When the domain of the input is known, a domain terminology may be employed to ensure specific phrases are present in a system’s predictions. Our goal in this work is to find a way to force the output of a model to contain such lexical constraints, while still taking advantage of the distribution learned from training data.
For Machine Translation (MT) usecases in particular, final translations are often produced by combining automatically translated output with user inputs. Examples include Post-Editing (PE) (Koehn, 2009; Specia, 2011) and InteractivePredictive MT (Foster, 2002; Barrachina et al., 2009; Green, 2014). These interactive scenarios can be unified by considering user inputs to be lexical constraints which guide the search for the optimal output sequence.
In this paper, we formalize the notion of lexical constraints, and propose a decoding algorithm which allows the specification of subsequences that are required to be present in a model’s output. Individual constraints may be single tokens or multi-word phrases, and any number of constraints may be specified simultaneously.
Although we focus upon interactive applications for MT in our experiments, lexically constrained decoding is relevant to any scenario where a model is asked to generate a sequence ŷ = {y0 . . . yT } given both an input x, and a set {c0...cn}, where each ci is a sub-sequence {ci0 . . . cij}, that must appear somewhere in ŷ. This makes our work applicable in a wide range of text generation scenarios, including image description, dialog generation, automatic summarization, and question answering.
The rest of this paper is organized as follows: Section 2 gives the necessary background for our
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Figure 1: A visualization of the decoding process for an actual example from our English-German MT experiments. The output token at each timestep appears at the top of the figure, with lexical constraints enclosed in boxes. Generation is shown in blue, Starting new constraints in green, and Continuing constraints in red. The function used to create the hypothesis at each timestep is written at the bottom. Each box in the grid represents a beam; a colored strip inside a beam represents an individual hypothesis in the beam’s k-best stack. Hypotheses with circles inside them are closed, all other hypotheses are open. (Best viewed in colour).
discussion of GBS, Section 3 discusses the lexically constrained decoding algorithm in detail, Section 4 presents our experiments, and Section 5 gives an overview of closely related work.
2 Background: Beam Search for Sequence Generation
Under a model parameterized by θ, let the best output sequence ŷ given input x be Eq. 1.
ŷ = argmax y∈{y[T]}
pθ(y|x), (1)
where we use {y[T]} to denote the set of all sequences of length T . Because the number of possible sequences for such a model is |v|T , where |v| is the number of output symbols, the search for ŷ can be made more tractable by factorizing pθ(y|x) into Eq. 2:
pθ(y|x) = T∏ t=0 pθ(yt|x; {y0 . . . yt−1}). (2)
The standard approach is thus to generate the output sequence from beginning to end, conditioning the output at each timestep upon the input x, and the already-generated symbols {y0 . . . yi−t}. However, greedy selection of the most probable output at each timestep, i.e.:
ŷt = argmax yi∈{v}
p(yi|x; {y0 . . . yt−1}), (3)
risks making locally optimal decisions which are actually globally sub-optimal. On the other hand, an exhaustive exploration of the output space would require scoring |v|T sequences, which is intractable for most real-world models. Thus, a search or decoding algorithm is often used as a compromise between these two extremes. A common solution is use a heuristic search to attempt
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Figure 2: Different structures for beam search. Boxes represent beams which hold k-best lists of hypotheses. (A) Chart Parsing using SCFG rules to cover spans in the input. (B) Source coverage as used in PB-SMT. (C) Sequence timesteps (as used in Neural Sequence Models), GBS is an extension of (C). In (A) and (B), hypotheses are finished once they reach the final beam. In (C), a hypothesis is only complete if it has generated an end-of-sequence (EOS) symbol.
to find the best output efficiently (Pearl, 1984; Koehn, 2010; Rush et al., 2013). The key idea is to discard bad options early, while avoiding discarding candidates that may be locally risky, but could eventually result in the best overall output.
Beam search (Och and Ney, 2004) is probably the most popular search algorithm for decoding sequences. Beam search is simple to implement, and is flexible in the sense that the semantics of the graph of beams can be adapted to take advantage of additional structure that may be available for specific tasks. For example, in Phrase-Based Statistical MT (PB-SMT) (Koehn, 2010), beams are organized by the number of source words that are covered by the hypotheses in the beam – a hypothesis is “finished” when it has covered all source words. In chart-based decoding algorithms such as CYK, beams are also tied to coverage of the input, but are organized as cells in a chart, which facilitates search for the optimal latent structure of the output (Chiang, 2007). Figure 2 visualizes three common ways to structure search. (A) and (B) depend upon explicit structural information between the input and output, (C) only assumes that the output is a sequence where later symbols depend upon earlier ones. Note also that (C) corresponds exactly to the bottom rows of Figures 1 and 3.
With the recent success of neural models for
text generation, beam search has become the de-facto choice for decoding optimal output sequences (Sutskever et al., 2014). However, with neural sequence models, we cannot organize beams by their explicit coverage of the input. A simpler alternative is to organize beams by output timesteps from t0 · · · tN , where N is a hyperparameter that can be set heuristically, for example by multiplying a factor with the length of the input to make an educated guess about the maximum length of the output (Sutskever et al., 2014). Output sequences are generally considered complete once a special “end-of-sentence”(EOS) token has been generated. Beam size in these models is also typically kept small, and recent work has shown that the performance of some architectures can actually degrade with larger beam size (Tu et al., 2016).
3 Grid Beam Search
Our goal is to organize decoding in such a way that we can constrain the search space to outputs which contain one or more pre-specified sub-sequences. We thus wish to use a model’s distribution both to “place” lexical constraints correctly, and to generate the parts of the output which are not covered by the constraints.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Algorithm 1 Pseudo-code for Grid Beam Search, note that t and c indices are 0-based 1: procedure CONSTRAINEDSEARCH(model, input, constraints, maxLen, numC, k) 2: startHyp⇐ model.getStartHyp(input, constraints) 3: Grid⇐ initGrid(maxLen, numC, k) . initialize beams in grid 4: Grid[0][0] = startHyp 5: for t = 1, t++, t < maxLen do 6: for c = max(0, (numC + t)−maxLen), c++, c ≤ min(t, numC) do 7: n, s, g = ∅ 8: for each hyp ∈ Grid[t− 1][c] do 9: if hyp.isOpen() then
10: g ⇐ g ⋃
model.generate(hyp, input, constraints) . generate new open hyps 11: end if 12: end for 13: if c > 0 then 14: for each hyp ∈ Grid[t− 1][c− 1] do 15: if hyp.isOpen() then 16: n⇐ n ⋃ model.start(hyp, input, constraints) . start new constrained hyps 17: else 18: s⇐ s ⋃ model.continue(hyp, input, constraints) . continue unfinished 19: end if 20: end for 21: end if 22: Grid[t][c] = k-argmax
h∈n ⋃ s ⋃ g model.score(h) . k-best scoring hypotheses stay on the beam
23: end for 24: end for 25: topLevelHyps⇐ Grid[:][numC] . get hyps in top-level beams 26: finishedHyps⇐ hasEOS(topLevelHyps) . finished hyps have generated the EOS token 27: bestHyp⇐ argmax
h∈finishedHyps model.score(h)
28: return bestHyp 29: end procedure
Algorithm 1 presents the pseudo-code for lexically constrained decoding, see Figures 1 and 3 for visualizations of the search process. Beams in the grid are indexed by t and c. The t variable tracks the timestep of the search, while the c variable indicates how many constraint tokens are covered by the hypotheses in the current beam. Note that each step of c covers a single constraint token. In other words, constraints is an array of sequences, where individual tokens can be indexed as constraintsij , i.e. tokenj in constrainti. The numC parameter in Algorithm 1 represents the total number of tokens in all constraints.
The hypotheses in a beam can be separated into two types (see lines 9-11 and 15-19 of Algorithm 1):
1. open hypotheses can either generate from the model’s distribution, or start available con-
straints,
2. closed hypotheses can only generate the next token for in a currently unfinished constraint.
At each step of the search the beam at Grid[t][c] is filled with candidates which may be created in three ways:
1. the open hypotheses in the beam to the left (Grid[t − 1][c]) may generate continuations from the model’s distribution pθ(yi|x, {y0 . . . yi−1}),
2. the open hypotheses in the beam to the left and below (Grid[t−1][c−1]) may start new constraints,
3. the closed hypotheses in the beam to the left and below (Grid[t−1][c−1]) may continue new constraints.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Therefore, the model in Algorithm 1 implements an interface with three functions: generate, start, and continue, which build new hypotheses in each of the three ways. Note that the scoring function of the model does not need to be aware of the existence of constraints, but it may be, for example via a feature which indicates if a hypothesis is part of a constraint or not.
The beams at the top level of the grid (beams where c = numConstraints) contain hypotheses which cover all of the constraints. Once a hypothesis on the top level generates the EOS token, it can be added to the set of finished hypotheses. The highest scoring hypothesis in the set of finished hypotheses is the best sequence which covers all constraints.
3.1 Multi-token Constraints
By distinguishing between open and closed hypotheses, we can allow for arbitrary multi-token phrases in the search. Thus, the set of constraints for a particular output may include both individual tokens and phrases. Each hypothesis also maintains a coverage vector ensuring that constraints cannot be repeated in a search path – hypotheses which have already covered constrainti can only generate, or start constraints that have not yet been covered.
Note also that discontinuous lexical constraints, such as phrasal verbs in English or German, are easy to incorporate into GBS, by adding filters to the search, which require that one or more conditions must be met before a constraint can be used. For example, adding the phrasal verb “ask 〈someone〉 out” as a constraint would mean using “ask” as constraint0 and “out” as constraint1, with two filters: one requiring that constraint1 cannot be used before constraint0, and another requiring that there must be at least one generated token between the constraints.
3.2 Subword Units
Both the computation of the score for a hypothesis, and the granularity of the tokens (character, subword, word, etc...) are left to the underlying model. Because our decoder can handle arbitrary constraints, there is a risk that constraints will contain tokens that were never observed in the training data, and thus are unknown by the model. Especially in domain adaptation scenarios, some userspecified constraints are very likely to contain unseen tokens. Subword representations provide an
elegant way to circumvent this problem, by breaking unknown or rare tokens into character n-grams which are part of the model’s vocabulary (Sennrich et al., 2016; Wu et al., 2016). In the experiments in Section 4, we use this technique to ensure that no input tokens are unknown, even if a constraint contains words which never appeared in the training data.
3.3 Efficiency
Because the number of beams is multiplied by the number of constraints, the runtime complexity of a naive implementation of GBS is O(ktc). Standard time-based beam search is O(kt); therefore, some consideration must be given to the efficiency of this algorithm. Note that the beams in each column c of Figure 3 are independent, meaning that GBS can be parallelized to allow all beams at each timestep to be filled simultaneously. Also, we find that the most time is spent computing the states for the hypothesis candidates, so by keeping the beam size small, we can make GBS significantly faster.
3.4 Models
The models used for our experiments are stateof-the-art Neural Machine Translation (NMT) systems using our own implementation of NMT with attention over the source sequence (Bahdanau et al., 2014). We used Blocks and Fuel to implement our NMT models (van Merrinboer et al., 2015). To conduct the experiments in the following section, we trained baseline translation models for English–German (EN-DE), English– French (EN-FR), and English–Portuguese (ENPT). We created a shared subword representation for each language pair by extracting a vocabulary of 80000 symbols from the concatenated source and target data. See the Appendix for more details on our training data and hyperparameter configuration for each language pair. The beamSize parameter is set to 10 for all experiments.
Because our experiments use NMT models, we can now be more explicit about the implementations of the generate, start, and continue functions for this GBS instantiation. For an NMT model at timestep t, generate(hypt−1) first computes a vector of output probabilities ot = softmax(g(yt−1, si, ci))
1 using the state information available from hypt−1. and returns the best k continuations, i.e. Eq. 4:
1we use the notation for the g function from Bahdanau et al. (2014)
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
gt = k-argmax i oti. (4)
The start and continue functions simply index into the softmax output of the model, selecting specific tokens instead of doing a k-argmax over the entire target language vocabulary. For example, to start constraint ci, we find the score of token ci0 , i.e. otci0 .
4 Experiments
4.1 Pick-Revise for Interactive Post Editing
Pick-Revise is an interaction cycle for MT PostEditing proposed by Cheng et al. (2016). Starting with the original translation hypothesis, a (simulated) user first picks a part of the hypothesis which is incorrect, and then provides the correct translation for that portion of the output. The userprovided correction is then used as a constraint for the next decoding cycle. The Pick-Revise process can be repeated as many times as necessary, with a new constraint being added at each cycle.
We modify the experiments of Cheng et al. (2016) slightly, and assume that the user only provides sequences of up to three words which are missing from the hypothesis2. To simulate user interaction, at each iteration we chose a phrase of up to three tokens from the reference translation which does not appear in the current MT hypotheses. In the strict setting, the complete phrase must be missing from the hypothesis. In the relaxed setting, only the first word must be missing. Table 1 shows results for a simulated editing session with four cycles. When a three-token phrase cannot be found, we backoff to two-token phrases, then to single tokens as constraints. If a hypothesis already matches the reference, no constraints are added. By specifying a new constraint of up to three words at each cycle, an increase of over 20 BLEU points is achieved in all language pairs.
4.2 Domain Adaptation via Terminology
The requirement for use of domain-specific terminologies is common in real-world applications of MT (Crego et al., 2016). Existing approaches incorporate placeholder tokens into NMT systems, which requires modifying the pre- and post- processing of the data, and training the system with
2NMT models do not use explicit alignment between source and target, so we cannot use alignment information to map target phrases to source phrases
data that contains the same placeholders which occur in the test data (Crego et al., 2016). The MT system also loses any possibility to model the tokens in the terminology, since they are represented by abstract tokens such as “〈TERM 1〉”. An attractive alternative is to simply provide term mappings as constraints, allowing any existing system to adapt to the terminology used in a new test domain.
For the target domain data, we use the Autodesk Post-Editing corpus (Zhechev, 2012), which is a dataset collected from actual MT post-editing sessions. The corpus is focused upon software localization, a domain which is likely to be very different from the WMT data used to train our general domain models. We divide the corpus into approximately 100,000 training sentences, and 1000 test segments, and automatically generate a terminology by computing the Pointwise Mutual Information (PMI) (Church and Hanks, 1990) between source and target n-grams in the training set. We extract all n-grams from length 2-5 as terminology candidates.
pmi(x;y) = log p(x, y)
p(x)p(y) (5)
npmi(x;y) = pmi(x;y)
h(x,y) (6)
Equations 5 and 6 show how we compute the normalized PMI for a terminology candidate pair. The PMI score is normalized to the range [−1,+1] by dividing by the entropy h of the joint probability p(x,y). We then filter the candidates to only include pairs whose PMI is≥ 0.9, and where both the source and target phrases occur at least five times in the corpus. When source phrases that match the terminology are observed in the test data, the corresponding target phrase is added to the constraints for that segment. Results are shown in Table 2.
This simple method of domain adaptation leads to a significant improvement in the BLEU score of the model without any human intervention. Thus, manually created domain terminologies are likely to lead to even greater performance gains. Surprisingly, even an automatically created terminology combined with the constrained decoding search yields performance improvements of approximately +2 BLEU points for En-De and EnFr, and a gain of almost 14 points for En-Pt. The large improvement for En-Pt is probably due to
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
ITERATION 0 1 2 3 Strict Constraints EN-DE 18.44 27.64 (+9.20) 36.66 (+9.01) 43.92 (+7.26) EN-FR 28.07 36.71 (+8.64) 44.84 (+8.13) 45.48 +(0.63) EN-PT* 15.41 23.54 (+8.25) 31.14 (+7.60) 35.89 (+4.75) Relaxed Constraints EN-DE 18.44 26.43 (+7.98) 34.48 (+8.04) 41.82 (+7.34) EN-FR 28.07 33.8 (+5.72) 40.33 (+6.53) 47.0 (+6.67) EN-PT* 15.41 23.22 (+7.80) 33.82 (+10.6) 40.75 (+6.93)
Table 1: Results for four simulated editing cycles using WMT test data. EN-DE uses newstest2013, EN-FR uses newstest2014, and EN-PT uses the Autodesk corpus discussed in Section 4.2. Improvement in BLEU score over the previous cycle is shown in parentheses. * indicates use of our test corpus created from Autodesk post-editing data.
the training data for this system being very different from the IT domain (see Appendix). Using a terminology with GBS is likely to benefit any model where the terminology used in the test domain is significantly different from the original training data.
System BLEU EN-DE Baseline 26.17 PMI Ngram Constraints 27.99 (+1.82) EN-FR Baseline 32.45 PMI Ngram Constraints 35.05 (+2.59) EN-PT Baseline 15.41 PMI Ngram Constraints 29.15 (+13.73)
Table 2: BLEU Results for EN-DE, EN-FR, and EN-PT terminology experiments using the Autodesk Post-Editing Corpus
4.3 Analysis
Subjective analysis of decoder output shows that phrases added as constraints are not only placed correctly within the output sequence, but also have global effects upon translation quality. This is a desirable effect for user interaction, since it implies that users can bootstrap quality by adding the most critical constraints (i.e. those that are most essential to the output), first. Table 3 shows several examples from the experiments in Table 1, where the addition of lexical constraints was able to guide our NMT systems away from initially quite low-scoring hypotheses to outputs which perfectly
match the reference translations.
5 Related Work
Most related work to date has presented modifications of SMT systems for specific usecases which constrain MT output via auxilliary inputs. The largest body of work considers Interactive Machine Translation (IMT): an MT system searches for the optimal target-language suffix given a complete source sentence and a desired prefix for the target output (Foster, 2002; Barrachina et al., 2009; Green, 2014). IMT can be viewed as subcase of constrained decoding, where there is only one constraint which is guaranteed to be placed at the beginning of the output sequence. Wuebker et al. (2016) introduce prefix-decoding, which modifies the SMT beam search to first ensure that the target prefix is covered, and only then continues to build hypotheses for the suffix using beams organized by coverage of the remaining phrases in the source segment.
Recently, some attention has also been given to SMT decoding with multiple lexical constraints. The Pick-Revise (PRIMT) (Cheng et al., 2016) framework for Interactive Post Editing introduces the concept of edit cycles. Translators specify constraints by editing a part of the MT output that is incorrect, and then asking the system for a new hypothesis, which must contain the user-provided correction. This process is repeated, maintaining constraints from previous iterations and adding new ones as needed. Importantly, their approach relies upon the phrase segmentation provided by the SMT system. The decoding algorithm can only make use of constraints that match phrase
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
EN-DE Source He was also an anti- smoking activist and took part in several campaigns . Original Hypothesis Es war auch ein Anti- Rauch- Aktiv- ist und nahmen an mehreren Kampagnen teil . Reference Constraints Ebenso setzte er sich gegen das Rauchen ein und nahm an mehreren Kampagnen teil . (1) Ebenso setzte er Constrained Hypothesis (2) gegen das Rauchen Ebenso setzte er sich gegen das Rauchen ein und nahm an mehreren Kampagnen teil . (3) nahm EN-FR Source At that point I was no longer afraid of him and I was able to love him . Original Hypothesis Je n’avais plus peur de lui et j’ètais capable de l’aimer . Reference Constraints Lá je n’ai plus eu peur de lui et j’ai pu l’aimer . (1) Lá je n’ai Constrained Hypothesis (2) j’ai pu Lá je n’ai plus eu peur de lui et j’ai pu l’aimer . (3) eu EN-PT Source Mo- dif- y drain- age features by selecting them individually . Original Hypothesis - Já temos as caracterı́sticas de extracção de idade , com eles individualmente . Reference Constraints Modi- fique os recursos de drenagem ao selec- ion- á-los individualmente . (1) drenagem ao selecConstrained Hypothesis (2) Modi- fique os Modi- fique os recursos de drenagem ao selec- ion- á-los individualmente . (3) recursos
Table 3: Manual analysis of examples from lexically constrained decoding experiments. “-” followed by whitespace indicates the internal segmentation of the translation model (see Section 3.2)
boundaries, because constraints are implemented as “rules” enforcing that source phrases must be translated as the aligned target phrases that have been selected as constraints. In contrast, our approach decodes at the token level, and is not dependent upon any explicit structure in the underlying model.
Domingo et al. (2016) also consider an interactive scenario where users first choose portions of an MT hypothesis to keep, then query for an updated translation which preserves these portions. The MT system decodes the source phrases which are not aligned to the user-selected phrases until the source sentence is fully covered. This approach is similar to the system of Cheng et al., and uses the “forced decoding” feature in Moses (Koehn et al., 2007).
Some recent work considers the inclusion of soft lexical constraints directly into deep models for dialog generation, and special cases, such as recipe generation from a list of ingredients (Wen et al., 2015; Kiddon et al., 2016). Such constraintaware models are complementary to our work, and could be used with GBS decoding without any change to the underlying models.
To the best of our knowledge, ours is the first work which considers general lexically constrained decoding for any model which outputs sequences, without relying upon alignments between input and output, and without using a search
organized by coverage of the input.
6 Conclusion
Lexically constrained decoding is a flexible way to incorporate arbitrary subsequences into the output of any model that generates output sequences token-by-token. A wide spectrum of popular text generation models have this characteristic, and GBS should be straightforward to use with any model that already uses beam search.
In translation interfaces where translators can provide corrections to an existing hypothesis, these user inputs can be used as constraints, generating a new output each time a user fixes an error. By simulating this scenario, we have shown that such a workflow can provide a large improvement in translation quality at each iteration.
By using a domain-specific terminology to generate target-side constraints, we have shown that a general domain model can be adapted to a new domain without any retraining. Surprisingly, this simple method can lead to significant performance gains, even when the terminology is created automatically.
In future work, we hope to evaluate GBS with models outside of MT, such as automatic summarization, image captioning or dialog generation. We also hope to introduce new constraintaware models, for example via secondary attention mechanisms over lexical constraints.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
A NMT System Configurations
We train all systems for 500000 iterations, with validation every 5000 steps. The best single model from validation is used in all of the experiments for a language pair. We use `2 regularization on all parameters with α = 1e−5. Dropout is used on the output layers with p(drop) = 0.5. We sort minibatches by source sentence length, and reshuffle training data after each epoch.
All systems use a bidirectional GRUs (Cho et al., 2014) to create the source representation and GRUs for the decoder transition. We use AdaDelta (Zeiler, 2012) to update gradients, and clip large gradients to 1.0.
Training Configurations EN-DE Embedding Size 300 Recurrent Layers Size 1000 Source Vocab Size 80000 Target Vocab Size 90000 Batch Size 50 EN-FR Embedding Size 300 Recurrent Layers Size 1000 Source Vocab Size 66000 Target Vocab Size 74000 Batch Size 40 EN-PT Embedding Size 200 Recurrent Layers Size 800 Source Vocab Size 60000 Target Vocab Size 74000 Batch Size 40
A.1 English-German Our English-German training corpus consists of 4.4 Million segments from the Europarl (Bojar et al., 2015) and CommonCrawl (Smith et al., 2013) corpora.
A.2 English-French Our English-French training corpus consists of 4.9 Million segments from the Europarl and CommonCrawl corpora.
A.3 English-Portuguese Our English-Portuguese training corpus consists of 28.5 Million segments from the Europarl, JRC-
Aquis (Steinberger et al., 2006) and OpenSubtitles3 corpora.
3http://www.opensubtitles.org/

This paper describes a straightforward extension to left-to-right beam search in order to allow it to incorporate lexical constraints in the form of word sequences that must appear in MT output. This algorithm is shown to be effective for interactive translation and domain adaptation.  Although the proposed extension is very simple, I think the paper makes a useful contribution by formalizing it. It is also interesting to know that NMT copes well with a set of unordered constraints having no associated alignment information. There seem to be potential applications for this technique beyond the ones investigated here, for example improving NMT  ability to handle non-compositional constructions, which is one of the few areas where it still might lag traditional SMT.  The main weakness of the paper is that the experiments are somewhat limited. The interactive MT simulation shows that the method basically works, but it is difficult to get a sense of how well - for instance, in how many cases the constraint was incorporated in an acceptable manner (the large BLEU score increases are only indirect evidence). Similarly, adaptation should have been  compared to the standard  fine-tuning d baseline, which would be relatively inexpensive to run on the 100K Autodesk corpus.  Despite this weakness, I think this is a decent contribution that deserves to be published.  Further details:  422 Given its common usage in PBMT,  coverage vector d is a potentially misleading term. The appropriate data structure seems more likely to be a coverage set.  Table 2 should also give some indication of the number of constraints per source sentence in the test corpora, to allow for calibration of the BLEU gains.