Knowledge as a Teacher: Knowledge-Guided Structural Attention Networks
Natural language understanding (NLU) is a core component of a dialogue system. Recently recurrent neural networks (RNN) obtained strong results on NLU due to their superior ability of preserving sequential information over time. Traditionally, the NLU module tags semantic slots for utterances considering their flat structures, as the underlying RNN structure is a linear chain. However, natural language exhibits linguistic properties that provide rich, structured information for better understanding. This paper introduces a novel model, knowledge-guided structural attention networks (K-SAN), a generalization of RNN to additionally incorporate nonflat network topologies guided by prior knowledge. There are two characteristics: 1) important substructures can be captured from small training data, allowing the model to generalize to previously unseen test data; 2) the model automatically figures out the salient substructures that are essential to predict the semantic tags of the given sentences, so that the understanding performance can be improved. The experiments on the benchmark ATIS data show that the proposed K-SAN architecture can effectively extract salient knowledge from substructures with an attention mechanism, and outperform the state-of-the-art neural network based frameworks.
1 Introduction
In the past decade, goal-oriented spoken dialogue systems (SDS), such as the virtual personal assistants Microsoft’s Cortana and Apple’s Siri, are
being incorporated in various devices and allow users to speak to systems freely in order to finish tasks more efficiently. A key component of these conversational systems is the natural language understanding (NLU) module-it refers to the targeted understanding of human speech directed at machines (Tur and De Mori, 2011). The goal of such “targeted” understanding is to convert the recognized user speech into a task-specific semantic representation of the user’s intention, at each turn, that aligns with the back-end knowledge and action sources for task completion. The dialogue manager then interprets the semantics of the user’s request and associated back-end results, and decides the most appropriate system action, by exploiting semantic context and user specific meta-information, such as geo-location and personal preferences (McTear, 2004; Rudnicky and Xu, 1999).
A typical pipeline of NLU includes: domain classification, intent determination, and slot filling (Tur and De Mori, 2011). NLU first decides the domain of user’s request given the input utterance, and based on the domain, predicts the intent and fills associated slots corresponding to a domain-specific semantic template. For example, Figure 1 shows a user utterance, “show me the flights from seattle to san francisco” and its semantic frame, find flight(origin=“seattle”, dest=“san francisco”). It is easy to see the relationship between the origin city and the destination city in this example, although these do not appear next to each other. Traditionally, domain detection and intent prediction are framed as utterance classification problems, where several classifiers such as support vector machines and maximum entropy have been employed (Haffner et al., 2003; Chelba et al., 2003; Chen et al., 2014). Then slot filling is framed as a word sequence tagging task, where the IOB (in-out-begin) format is ap-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
show me the flights from seattle to san francisco
O O O O B-origin O B-dest I-destO
Figure 1: An example utterance annotated with its semantic slots in the IOB format (S).
plied for representing slot tags as illustrated in Figure 1, and hidden Markov models (HMM) or conditional random fields (CRF) have been employed for slot tagging (Pieraccini et al., 1992; Wang et al., 2005).
With the advances on deep learning, deep belief networks (DBNs) with deep neural networks (DNNs) have been applied to domain and intent classification tasks (Sarikaya et al., 2011; Tur et al., 2012; Sarikaya et al., 2014). Recently, Ravuri and Stolcke (2015) proposed an RNN architecture for intent determination. For slot filling, deep learning has been viewed as a feature generator and the neural architecture can be merged with CRFs (Xu and Sarikaya, 2013). Yao et al. (2013) and Mesnil et al. (2015) later employed RNNs for sequence labeling in order to perform slot filling. However, the above studies benefit from large training data without leveraging any existing knowledge. When tagging sequences RNNs consider them as flat structures, with their underlying linear chain structures, potentially ignoring the structured information typical of natural language sequences.
Hierarchical structures and semantic relationships contain linguistic characteristics of input word sequences forming sentences, and such information may help interpret their meaning. Furthermore, prior knowledge would help in the tagging of sequences, especially when dealing with previously unseen sequences (Tur et al., 2010; Deoras and Sarikaya, 2013). Prior work exploited external web-scale knowledge graphs such as Freebase and Wikipedia for improving NLU (Heck et al., 2013; Ma et al., 2015b; Chen et al., 2014) Liu et al. (2013) and Chen et al. (2015) proposed approaches that leverage linguistic knowledge encoded in parse trees for language understanding, where the extracted syntactic structural features and semantic dependency features enhance inference model learning, and the model achieves better language understanding performance in various domains.
Even with the emerging paradigm of integrating deep learning and linguistic knowledge for differ-
ent NLP tasks (Socher et al., 2014), most of the previous work utilized such linguistic knowledge and knowledge bases as additional features as input to neural networks, and then learned the models for tagging sequences. These feature enrichment based approaches have some possible limitations: 1) poor generalization and 2) error propagation. Poor generalization comes from the mismatch between knowledge bases and the input data, and then the incorrectly extracted features due to errors in previous processing propagate errors to the neural models. In order to address the issues and better learn the sequence tagging models, this paper proposes knowledge-guided structural attention networks, K-SAN, a generalization of RNNs that automatically learn the attention guided by external or prior knowledge and generate sentence-based representations specifically for modeling sequence tagging. The main difference between K-SAN and previous approaches is that knowledge plays the role of a teacher to guide networks where and how much to focus attention considering the whole linguistic structure simultaneously. Our main contributions are three-fold: • End-to-end learning
To our knowledge, this is the first neural network approach that utilizes general knowledge as guidance in an end-to-end fashion, where the model automatically learns important substructures with an attention mechanism. • Generalization for different knowledge
There is no required schema of knowledge, and different types of parsing results, such as dependency relations, knowledge graphspecific relations, and parsing output of handcrafted grammars, can serve as the knowledge guidance in this model. • Efficiency and parallelizability
Because the substructures from the input utterance are modeled separately, modeling time may not increase linearly with respect to the number of words in the input sentence.
In the following sections, we empirically show the benefit of K-SAN on the targeted NLU task.
2 Related Work
Knowledge-Based Representations There is an emerging trend of learning representations at different levels, such as word embeddings (Mikolov et al., 2013), character
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
embeddings (Ling et al., 2015), and sentence embeddings (Le and Mikolov, 2014; Huang et al., 2013). In addition to fully unsupervised embedding learning, knowledge bases have been widely utilized to learn entity embeddings with specific functions or relations (Celikyilmaz and Hakkani-Tur, 2015; Yang et al., 2014). Different from prior work, this paper focuses on learning composable substructure embeddings that are informative for understanding.
Recently linguistic structures are taken into account in the deep learning framework. Ma et al. (2015a) and Tai et al. (2015) both proposed dependency-based approaches to combine deep learning and linguistic structures, where the model used tree-based n-grams instead of surface ones to capture knowledge-guided relations for sentence modeling and classification. Roth and Lapata (2016) utilized lexicalized dependency paths to learn embedding representations for semantic role labeling. However, the performance of these approaches highly depends on the quality of “whole” sentence parsing, and there is no control of degree of attentions on different substructures. Learning robust representations incorporating whole structures still remains unsolved. In this paper, we address the limitation by proposing K-SAN to learn robust representations of whole sentences, where the whole representation is composed of the salient substructures in order to avoid error propagation.
Neural Attention and Memory Model One of the earliest work with a memory component applied to language processing is memory networks (Weston et al., 2015; Sukhbaatar et al., 2015), which encode facts into vectors and store them in the memory for question answering (QA). Following their success, Xiong et al. (2016) proposed dynamic memory networks (DMN) to additionally capture position and temporality of transitive reasoning steps for different QA tasks. The idea is to encode important knowledge and store it into memory for future usage with attention mechanisms. Attention mechanisms allow neural network models to selectively pay attention to specific parts. There are also various tasks showing the effectiveness of attention mechanisms.
However, most previous work focused on the classification or prediction tasks (predicting a single word given a question), and there are few studies for NLU tasks (slot tagging). Based on the
fact that the linguistic or knowledge-based substructures can be treated as prior knowledge to benefit language understanding, this work borrows the idea from memory models to improve NLU. Unlike the prior NLU work that utilized representations learned from knowledge bases to enrich features of the current sentence, this paper directly learns a sentence representation incorporating memorized substructures with an automatically decided attention mechanism in an end-toend manner.
3 Knowledge-Guided Structural Attention Networks (K-SAN)
Given an utterance with a sequence of words/tokens ~s = w1, ..., wT , our NLU model is to predict corresponding semantic tags ~y = y1, ..., yT for each word/token by incorporating knowledge-guided structures. The proposed model is illustrated in Figure 2. The knowledge encoding module first leverages external knowledge to generate a linguistic structure for the utterance, where a discrete set of knowledge-guided substructures {xi} is encoded into a set of vector representations (§ 3.1). The model learns the representation for the whole sentence by paying different attention on the substructures (§ 3.2). Then the learned vector encoding the knowledge-guided structure is used for improving the semantic tagger (§ 4).
3.1 Knowledge Encoding Module
The prior knowledge obtained from external resources, such as dependency relations, knowledge bases, etc., provides richer information to help decide the semantic tags given an input utterance. This paper takes dependency relations as an example for knowledge encoding, and other structured relations can be applied in the same way. The input utterance is parsed by a dependency parser, and the substructures are built according to the paths from the root to all leaves (Chen and Manning, 2014). For example, the dependency parsing of the utterance “show me the flights from seattle to san francisco” is shown in Figure 3, where the associated substructures are obtained from the parsing tree for knowledge encoding. Here we do not utilize the dependency relation labels in the experiments for better generalization, because the labels may not be always available for different knowledge resources. Note that the number of substruc-
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
knowledge-guided structure {xi}
Knowledge Encoder
Sentence Encoder
Inner Product
u
mi
Knowledge Attention Distributionpi
Encoded Knowledge Representation
Weighted Sum
∑
h
o
Knowledge-Guided Representation
slot tagging sequence
s
y
show me the flights from seattle to san francisco
ROOT
Input Sentence
ht-1 ht+1ht W W W W
wt-1
yt-1
U
Mwt
U
wt+1
U
V
yt
V
yt+1
V
MM
RNN Tagger
Knowledge Encoding Module
Mkg
Min Mout
“show me the flights from seattle to san francisco”
Figure 2: The illustration of knowledge-guided structural attention networks (K-SAN) for NLU.
show
me
the
flights
from
seattle
to
san
francisco
ROOT
1.
3.
4.
2. 1. show me
2. show flights the 3. show flights from seattle 4. show flights to francisco san
Sentence s
show me the flights from seattle to san francisco
Knowledge-Guided Substructure xi
Figure 3: The knowledge-guided substructures of dependency parsing on an example sentence.
tures may be less than the number of words in the utterance, because non-leaf nodes do not have corresponding substructure in order to reduce the duplicated information in the model. The top-left component of Figure 2 illustrates the module for modeling knowledge-guided substructures.
3.2 Model Architecture
The model embeds all knowledge-guided substructures into a continuous space and stores embeddings of all x’s in the knowledge memory. The representation of the input utterance is then compared with encoded knowledge representations to integrate the carried structure guided by knowledge via an attention mechanism. Then the knowledge-guided representation of the sentence is taken together with the word sequence for estimating the semantic tags. Four main procedures are described below.
Encoded Knowledge Representation To store the knowledge-guided structure, we convert each substructure (e.g. path starting from the root to the leaf in the dependency tree), xi, into a structure vector mi with dimension d by embedding
the substructure in a continuous space through the knowledge encoding network Mkg. The input utterance s is also embedded to a vector u with the same dimension through the network model Min. We apply the three types for knowledge encoding models, Mkg and Min, in order to model multiple words from a substructure xi or an input sentence s into a vector representation: 1) fully-connected neural networks (NN) with linear activation, 2) recurrent neural networks (RNN), and 3) convolutional neural networks (CNN) with a window size 3 and a max-pooling operation. For example, one of substructures shown in Figure 3, “show flights seattle from”, is encoded into a vector embedding. In the experiments, the weights of Mkg and Min are tied together based on their consistent ability of sequence encoding.
Knowledge Attention Distribution In the embedding space, we compute the match between the current utterance vector u and its substructure vector mi by taking their inner product followed by a softmax.
pi = softmax(uTmi), (1)
where pi can be viewed as attention distribution for modeling important substructures from external knowledge in order to understand the current utterance.
Sentence Representation In order to encode the knowledge-guided structure, a vector h is a sum over the encoded knowledge embeddings weighted by the attention distribution.
h = ∑ i pimi, (2)
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
which indicates that the sentence pays different attention to different substructures guided from external knowledge. Because the function from input to output is smooth, we can easily compute gradients and back propagate through it. Then the sum of the substructure vector h and the current input embedding u are then passed through a neural network model Mout to generate an output knowledge-guided representation o,
o =Mout(h+ u), (3)
where we employ a fully-connected dense network for Mout.
Sequence Tagging To estimate the tag sequence ~y corresponding to an input word sequence ~s, we use an RNN module for training a slot tagger, where the knowledge-guided representation o is fed into the input of the model in order to incorporate the structure information.
~y = RNN(o,~s) (4)
4 Recurrent Neural Network Tagger
4.1 Chain-Based RNN Tagger
Given ~s = w1, ..., wT , the model is to predict ~y = y1, ..., yT where the tag yi is aligned with the word wi. We use the Elman RNN architecture, consisting of an input layer, a hidden layer, and an output layer (Elman, 1990). The input, hidden and output layers consist of a set of neurons representing the input, hidden, and output at each time step t, wt, ht, and yt, respectively.
ht = φ(Wwt + Uht−1), (5)
ŷt = softmax(V ht), (6)
where φ is a smooth bounded function such as tanh, and ŷt is the probability distribution over of semantic tags given the current hidden state ht. The sequence probability can be formulated as
p(~y | ~s) = ∏ i p(yi | w1, ..., wi). (7)
The model can be trained using backpropagation to maximize the conditional likelihood of the training set labels.
To overcome the frequent vanishing gradients issue when modeling long-term dependencies, gated RNN was designed to use a more sophisticated activation function, such as long shortterm memory (LSTM) and gated recurrent unit
(GRU) (Hochreiter and Schmidhuber, 1997; Cho et al., 2014). RNNs employing either of these recurrent units have been shown to perform well in tasks that require capturing long-term dependencies (Mesnil et al., 2015; Yao et al., 2014; Graves et al., 2013; Sutskever et al., 2014). This paper applies RNN with GRU cells to allow each recurrent unit to adaptively capture dependencies of different time scales (Cho et al., 2014; Chung et al., 2014), because RNN-GRU can yield comparable performance as RNN-LSTM with need of fewer parameters and less data for generalization (Chung et al., 2014; Jozefowicz et al., 2015).
4.2 Knowledge-Guided RNN Tagger
In order to model the encoded knowledge from previous turns, for each time step t, the knowledge-guided sentence representation o in (3) is fed into the RNN model together with the word wt. For the plain RNN, the hidden layer can be formulated as
ht = φ(Mo+Wwt + Uht−1) (8)
to replace (5) as illustrated in the right block of Figure 2. RNN-GRU can incorporate the encoded knowledge in the similar way, where Mo can be added into gating mechanisms for modeling contextual knowledge similarly.
4.3 Joint RNN Tagger
Because the chain-based tagger and the knowledge-guided tagger carry different information, the joint RNN tagger is proposed to balance the information between two model architectures. Figure 4 presents the architecture of
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
the joint RNN tagger.
h1t = φ(W 1wt + U 1ht−1), (9) h2t = φ(Mo+W 2wt + U 2ht−1), (10)
ŷt = softmax(V (αh1t + (1− α)h2t )),(11)
where α is the weight for balancing chainbased and knowledge-guided information. By jointly considering chain-based information (h1t ) and knowledge-guided information (h2t ), the joint RNN tagger is expected to achieve better generalization, and the performance may be less sensitive to poor structures from external knowledge. In the experiments, α is set to 0.5 for balancing two sides. The objective of the proposed model is to maximize the sequence probability p(~y | ~s) in (7), and the model can be trained in an endto-end manner, where the error would be backpropagated through the whole architecture.
5 Experiments
5.1 Experimental Setup
The dataset for experiments is the benchmark ATIS corpus, which is extensively used by the NLU community (Mesnil et al., 2015). There are 4978 training utterances selected from Class A (context independent) in the ATIS-2 and ATIS-3, while there are 893 utterances selected from the ATIS-3 Nov93 and Dec94. In the experiments, we only use lexical features. In order to show the robustness to data scarcity, we conduct the experiments with 3 different sizes of training data (Small, Medium, and Large). The evaluation metrics for NLU is F-measure on the predicted slots.
For experiments with K-SAN, we parse all data with the Stanford dependency parser (Chen and Manning, 2014) and represent words as their embeddings trained on the in-domain data, where the parser is pre-trained on PTB. The loss function is cross-entropy, and the optimizer we use is adam with the default setting (Kingma and Ba, 2014), where the learning rate λ = 0.001, β1 = 0.9, β2 = 0.999, and = 1e−08. The maximum iteration for training our K-SAN models is set as 300. The dimensionality of input word embeddings is 100, and the hidden layer sizes are in {50, 100, 150}. The dropout rates are set as {0.25, 0.50}. All reported results are from the joint RNN tagger, and the hyperparameters are tuned in the dev set for all experiments.
5.2 Baseline
To validate the effectiveness of the proposed model, we compare the performance with the following baselines. • Baseline:
– CRF Tagger (Tur et al., 2010): predicts a semantic slot for each word with a context window (size = 5). – RNN Tagger (Mesnil et al., 2015): predicts a semantic slot for each word. – CNN Encoder-Tagger (Kim, 2014): tag semantic slots with consideration of sentence embeddings learned by a convolutional model.
• Structural: The NLU models utilize linguistic information when tagging slots, where DCNN and Tree-RNN are the state-of-theart approaches for embedding sentences with linguistic structures.
– CRF Tagger (Tur et al., 2010): predicts slots based on the lexical (5-word window) and syntactic (dependent head in the parsing tree) features. – DCNN (Ma et al., 2015a): predicts slots by incorporating sentence embeddings learned by a convolutional model with consideration of dependency tree structures. – Tree-RNN (Tai et al., 2015): predicts slots with sentence embeddings learned by an RNN model based on the tree structures of sentences.
5.3 Slot Filling Results
Table 1 shows the performance of slot filling on different size of training data, where there are three datasets (Small, Medium, and Large use 1/40, 1/10, and whole training data). For baselines (models without knowledge features), CNN Encoder-Tagger achieves the best performance on all datasets.
Among structural models (models with knowledge encoding), Tree-RNN Encoder-Tagger performs better for Small data but slightly worse than the DCNN Encoder-Tagger.
CNN (Kim, 2014) performs better compared to DCNN (Ma et al., 2015a) and Tree-RNN (Tai et al., 2015), even though CNN does not leverage external knowledge when encoding sentences. When comparing the NLU performance between baselines and other state-of-the-art structural mod-
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Model Dataset Encoder (Mkg/Min) Knowledge Tagger Small Medium Large
Baseline - 7 CRF 58.94 78.74 89.73 - 7 RNN 68.58 84.55 92.97 CNN 7 RNN 73.57 85.52 93.88 Structural - 3 CRF 59.55 78.71 90.13 DCNN 3 RNN 70.24 83.80 93.25 Tree-RNN 3 RNN 73.50 83.92 92.28 Proposed K-SAN (NN) 3 RNN 74.11† 85.97 93.98†
K-SAN (RNN) 3 RNN 73.13 86.85† 94.97† K-SAN (CNN) 3 RNN 74.60† 87.99† 94.86†
Table 1: The F1 scores of predicted slots on the different size of ATIS training examples, where K-SAN utilizes the dependency relations parsed from the Stanford parser. Small: 1/40 set; Medium: 1/10 set; Large: original set. († indicates that the performance is significantly better than all baseline models with p < 0.05 in the t-test.)
find nonstop flights from salt lake city to new york on saturday april ninth
find nonstop flights from salt lake city to new york on saturday april ninth
find nonstop flights from salt lake city to new york on saturday april ninth
flight_stop fromloc.city_name toloc.city_name depart_date. day_name depart_date. month_name depart_date. day_number
Small
Medium
Large
Figure 5: The visualization of the decoded knowledge-guided structural attention for both relations and words learned from different size of training data. Relations and words with darker color indicate higher attention weights generated by the proposed K-SAN with CNN. The slot tags are shown in the figure for reference. Note that the dependency relations are incorrectly parsed by the Stanford parser in this example, but our model is still able to benefit from the structural information.
els, there is no significant difference. This suggests that encoding sentence information without distinguishing substructure may not capture salient semantics in order to improve understanding performance.
Among the proposed K-SAN models, CNN for encoding performs best on Small (75% on F1) and Medium (88% on F1), and RNN for encoding performs best on the Large set (95% on F1). Also, most of the proposed models outperform all baselines, where the improvement for the small dataset is more significant. This suggests that the proposed models carry better generalization and are less sensitive to unseen data. For example, given an utterance “which flights leave on monday from montreal and arrive in chicago in the morning”, “morning” can be correctly tagged with a semantic tag B-arrive time.period of day by K-SAN, but it is incorrectly tagged with Bdepart time.period of day by baselines, because knowledge guides the model to pay correct atten-
tion to salient substructures. The proposed model presents the state-of-the-art performance on the large dataset, showing the effectiveness of leveraging knowledge-guided structures for learning embeddings that can be used for specific tasks and the robustness to data scarcity and mismatch.
5.4 Attention Analysis
In order to show the effectiveness of boosting performance by learning correct attention from much smaller training data through the proposed model, we present the visualization of the attention for both words and relations decoded by K-SAN with CNN in the Figure 5. The darker color of blocks and lines indicates the higher attention for words and relations respectively. From the figure, the words and the relations with higher attention are the most crucial parts for predicting correct slots, e.g. origin, destination, and time. Furthermore, the difference of attention distribution between three datasets is not significant; this suggests that
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Approach Knowledge (Max #Substructure) Small Medium Large
CRF Syntax: Dependency Tree
Stanford - 59.55 78.71 90.13 SyntaxNet - 61.09 78.87 90.92
Semantics: AMR Graph Rule-Based - 59.55 79.15 89.97 JAMR - 61.12 78.64 90.25
K-SAN (CNN) Syntax: Dependency Tree
Stanford 53 74.60 87.99 94.86 SyntaxNet 25 74.35 88.40 95.00
Semantics: AMR Graph Rule-Based 19 74.32 88.14 94.85 JAMR 8 74.27 88.27 94.89
Table 2: The F1 scores of predicted slots with knowledge from different resources.
Sentence s show me the flights from seattle to san francisco
Knowledge-Guided Substructure xi
1. show you 2. show flight seattle 3. show flight san francisco 4. show i
show
you
flight I
1.
2.
4.
city
city
Seattle
San Francisco 3.
Figure 6: The knowledge-guided substructures of the AMR graph.
our proposed model is able to pay correct attention to important substructures guided by the external knowledge even the training data is scarce.
5.5 Knowledge Generalization
In order to show the capacity of generalization to different knowledge resources, we perform the KSAN model for different knowledge bases. Below we compare two types of knowledge formats: dependency tree and Abstract Meaning Representation (AMR). AMR is a semantic formalism in which the meaning of a sentence is encoded as a rooted, directed, acyclic graph (Banarescu et al., 2013), where nodes represent concepts, and labeled directed edges represent the relations between two concepts. The formalism is based on propositional logic and neo-Davidsonian event representations (Parsons, 1990; Davidson, 1967). The semantic concepts in AMR were leveraged to benefit multiple NLP tasks (Liu et al., 2015). Unlike syntactic information from dependency trees, the AMR graph contains semantic information, which may offer more specific conceptual relations. Figure 6 shows an AMR graph associated with the example utterance as same as one in Figure 3 and how the knowledge-guided substructures are constructed.
Table 2 presents the performance of CRF and K-SAN with CNN taggers that utilize dependency relations and AMR edges as knowledge guidance on the same datasets, where CRF takes the
head words from either dependency trees or AMR graphs as additional features and K-SAN incorporates knowledge-guided substructures. The dependency trees are obtained from the Stanford dependency parser or the SyntaxNet parser1, and AMR graphs are generated by a rule-based AMR parser or JAMR2.
Among four knowledge resources (different types and obtained from different parsers), all results show the similar performance for three sizes of datasets. The maximum number of substructures for the dependency tree is larger than the number in the AMR graph (53 and 25 v.s. 19 and 8), because syntax is more general and may provide richer cues for guiding more attention while semantics is more specific and may offer stronger guidance. In sum, the models applying four different resources achieve similar performance, and all significantly outperform the state-of-the-art NLU tagger, showing the effectiveness, generalization, and robustness of the proposed K-SAN model.
6 Conclusion
This paper proposes a novel model, knowledgeguided structural attention networks (K-SAN), that leverages prior knowledge as guidance to incorporate non-flat topologies and learn suitable attention for different substructures that are salient for specific tasks. The structured information can be captured from small training data, so the model has better generalization and robustness. The experiments show benefits and effectiveness of the proposed model on the language understanding task, where all knowledge-guided substructures captured by different resources help tagging performance, and the state-of-the-art performance is achieved on the ATIS benchmark dataset.
1https://github.com/tensorflow/models/ tree/master/syntaxnet
2https://github.com/jflanigan/jamr
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
This paper proposes a neural network architecture that represent structural linguistic knowledge in a memory network for sequence tagging tasks (in particular, slot-filling of the natural language understanding unit in conversation systems). Substructures (e.g. a node in the parse tree) is encoded as a vector (a memory slot) and a weighted sum of the substructure embeddings are fed in a RNN at each time step as additional context for labeling.  -----Strengths-----  I think the main contribution of this paper is a simple way to  flatten  structured information to an array of vectors (the memory), which is then connected to the tagger as additional knowledge. The idea is similar to structured   syntax-based attention (i.e. attention over nodes from treeLSTM); related work includes Zhao et al on textual entailment, Liu et al. on natural language inference, and Eriguchi et al. for machine translation. The proposed substructure encoder is similar to DCNN (Ma et al.): each node is embedded from a sequence of ancestor words. The architecture does not look entirely novel, but I kind of like the simple and practical approach compared to prior work.  -----Weaknesses-----  I'm not very convinced by the empirical results, mostly due to the lack of details of the baselines. Comments below are ranked by decreasing importance.  -  The proposed model has two main parts: sentence embedding and substructure embedding. In Table 1, the baseline models are TreeRNN and DCNN, they are originally used for sentence embedding but one can easily take the node substructure embedding from them too. It's not clear how they are used to compute the two parts.  - The model uses two RNNs: a chain-based one and a knowledge guided one. The only difference in the knowledge-guided RNN is the addition of a  knowledge  vector from the memory in the RNN input (Eqn 5 and 8). It seems completely unnecessary to me to have separate weights for the two RNNs. The only advantage of using two is an increase of model capacity, i.e. more parameters. Furthermore, what are the hyper-parameters   size of the baseline neural networks? They should have comparable numbers of parameters.  - I also think it is reasonable to include a baseline that just input additional knowledge as features to the RNN, e.g. the head of each word, NER results etc.  - Any comments   results on the model's sensitivity to parser errors?  Comments on the model:  - After computing the substructure embeddings, it seems very natural to compute an attention over them at each word. Is there any reason to use a static attention for all words? I guess as it is, the  knowledge  is acting more like a filter to mark important words. Then it is reasonable to include the baseline suggest above, i.e. input additional features.  - Since the weight on a word is computed by inner product of the sentence embedding and the substructure embedding, and the two embeddings are computed by the same RNN CNN, doesn't it means nodes   phrases similar to the whole sentence gets higher weights, i.e. all leaf nodes?  - The paper claims the model generalizes to different knowledge but I think the substructure has to be represented as a sequence of words, e.g. it doesn't seem straightforward for me to use constituent parse as knowledge here.  Finally, I'm hesitating to call it  knowledge . This is misleading as usually it is used to refer to world   external knowledge such as a knowledge base of entities, whereas here it is really just syntax, or arguably semantics if AMR parsing is used.  -----General Discussion-----  This paper proposes a practical model which seems working well on one dataset, but the main ideas are not very novel (see comments in Strengths). I think as an ACL paper there should be more takeaways. More importantly, the experiments are not convincing as it is presented now. Will need some clarification to better judge the results.  -----Post-rebuttal-----  The authors did not address my main concern, which is whether the baselines (e.g. TreeRNN) are used to compute substructure embeddings independent of the sentence embedding and the joint tagger. Another major concern is the use of two separate RNNs which gives the proposed model more parameters than the baselines. Therefore I'm not changing my scores.