Knowledge as a Teacher: Knowledge-Guided Structural Attention Networks
Natural language understanding (NLU) is a core component of a dialogue system. Recently recurrent neural networks (RNN) obtained strong results on NLU due to their superior ability of preserving sequential information over time. Traditionally, the NLU module tags semantic slots for utterances considering their flat structures, as the underlying RNN structure is a linear chain. However, natural language exhibits linguistic properties that provide rich, structured information for better understanding. This paper introduces a novel model, knowledge-guided structural attention networks (K-SAN), a generalization of RNN to additionally incorporate nonflat network topologies guided by prior knowledge. There are two characteristics: 1) important substructures can be captured from small training data, allowing the model to generalize to previously unseen test data; 2) the model automatically figures out the salient substructures that are essential to predict the semantic tags of the given sentences, so that the understanding performance can be improved. The experiments on the benchmark ATIS data show that the proposed K-SAN architecture can effectively extract salient knowledge from substructures with an attention mechanism, and outperform the state-of-the-art neural network based frameworks.
1 Introduction
In the past decade, goal-oriented spoken dialogue systems (SDS), such as the virtual personal assistants Microsoft’s Cortana and Apple’s Siri, are
being incorporated in various devices and allow users to speak to systems freely in order to finish tasks more efficiently. A key component of these conversational systems is the natural language understanding (NLU) module-it refers to the targeted understanding of human speech directed at machines (Tur and De Mori, 2011). The goal of such “targeted” understanding is to convert the recognized user speech into a task-specific semantic representation of the user’s intention, at each turn, that aligns with the back-end knowledge and action sources for task completion. The dialogue manager then interprets the semantics of the user’s request and associated back-end results, and decides the most appropriate system action, by exploiting semantic context and user specific meta-information, such as geo-location and personal preferences (McTear, 2004; Rudnicky and Xu, 1999).
A typical pipeline of NLU includes: domain classification, intent determination, and slot filling (Tur and De Mori, 2011). NLU first decides the domain of user’s request given the input utterance, and based on the domain, predicts the intent and fills associated slots corresponding to a domain-specific semantic template. For example, Figure 1 shows a user utterance, “show me the flights from seattle to san francisco” and its semantic frame, find flight(origin=“seattle”, dest=“san francisco”). It is easy to see the relationship between the origin city and the destination city in this example, although these do not appear next to each other. Traditionally, domain detection and intent prediction are framed as utterance classification problems, where several classifiers such as support vector machines and maximum entropy have been employed (Haffner et al., 2003; Chelba et al., 2003; Chen et al., 2014). Then slot filling is framed as a word sequence tagging task, where the IOB (in-out-begin) format is ap-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
show me the flights from seattle to san francisco
O O O O B-origin O B-dest I-destO
Figure 1: An example utterance annotated with its semantic slots in the IOB format (S).
plied for representing slot tags as illustrated in Figure 1, and hidden Markov models (HMM) or conditional random fields (CRF) have been employed for slot tagging (Pieraccini et al., 1992; Wang et al., 2005).
With the advances on deep learning, deep belief networks (DBNs) with deep neural networks (DNNs) have been applied to domain and intent classification tasks (Sarikaya et al., 2011; Tur et al., 2012; Sarikaya et al., 2014). Recently, Ravuri and Stolcke (2015) proposed an RNN architecture for intent determination. For slot filling, deep learning has been viewed as a feature generator and the neural architecture can be merged with CRFs (Xu and Sarikaya, 2013). Yao et al. (2013) and Mesnil et al. (2015) later employed RNNs for sequence labeling in order to perform slot filling. However, the above studies benefit from large training data without leveraging any existing knowledge. When tagging sequences RNNs consider them as flat structures, with their underlying linear chain structures, potentially ignoring the structured information typical of natural language sequences.
Hierarchical structures and semantic relationships contain linguistic characteristics of input word sequences forming sentences, and such information may help interpret their meaning. Furthermore, prior knowledge would help in the tagging of sequences, especially when dealing with previously unseen sequences (Tur et al., 2010; Deoras and Sarikaya, 2013). Prior work exploited external web-scale knowledge graphs such as Freebase and Wikipedia for improving NLU (Heck et al., 2013; Ma et al., 2015b; Chen et al., 2014) Liu et al. (2013) and Chen et al. (2015) proposed approaches that leverage linguistic knowledge encoded in parse trees for language understanding, where the extracted syntactic structural features and semantic dependency features enhance inference model learning, and the model achieves better language understanding performance in various domains.
Even with the emerging paradigm of integrating deep learning and linguistic knowledge for differ-
ent NLP tasks (Socher et al., 2014), most of the previous work utilized such linguistic knowledge and knowledge bases as additional features as input to neural networks, and then learned the models for tagging sequences. These feature enrichment based approaches have some possible limitations: 1) poor generalization and 2) error propagation. Poor generalization comes from the mismatch between knowledge bases and the input data, and then the incorrectly extracted features due to errors in previous processing propagate errors to the neural models. In order to address the issues and better learn the sequence tagging models, this paper proposes knowledge-guided structural attention networks, K-SAN, a generalization of RNNs that automatically learn the attention guided by external or prior knowledge and generate sentence-based representations specifically for modeling sequence tagging. The main difference between K-SAN and previous approaches is that knowledge plays the role of a teacher to guide networks where and how much to focus attention considering the whole linguistic structure simultaneously. Our main contributions are three-fold: • End-to-end learning
To our knowledge, this is the first neural network approach that utilizes general knowledge as guidance in an end-to-end fashion, where the model automatically learns important substructures with an attention mechanism. • Generalization for different knowledge
There is no required schema of knowledge, and different types of parsing results, such as dependency relations, knowledge graphspecific relations, and parsing output of handcrafted grammars, can serve as the knowledge guidance in this model. • Efficiency and parallelizability
Because the substructures from the input utterance are modeled separately, modeling time may not increase linearly with respect to the number of words in the input sentence.
In the following sections, we empirically show the benefit of K-SAN on the targeted NLU task.
2 Related Work
Knowledge-Based Representations There is an emerging trend of learning representations at different levels, such as word embeddings (Mikolov et al., 2013), character
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
embeddings (Ling et al., 2015), and sentence embeddings (Le and Mikolov, 2014; Huang et al., 2013). In addition to fully unsupervised embedding learning, knowledge bases have been widely utilized to learn entity embeddings with specific functions or relations (Celikyilmaz and Hakkani-Tur, 2015; Yang et al., 2014). Different from prior work, this paper focuses on learning composable substructure embeddings that are informative for understanding.
Recently linguistic structures are taken into account in the deep learning framework. Ma et al. (2015a) and Tai et al. (2015) both proposed dependency-based approaches to combine deep learning and linguistic structures, where the model used tree-based n-grams instead of surface ones to capture knowledge-guided relations for sentence modeling and classification. Roth and Lapata (2016) utilized lexicalized dependency paths to learn embedding representations for semantic role labeling. However, the performance of these approaches highly depends on the quality of “whole” sentence parsing, and there is no control of degree of attentions on different substructures. Learning robust representations incorporating whole structures still remains unsolved. In this paper, we address the limitation by proposing K-SAN to learn robust representations of whole sentences, where the whole representation is composed of the salient substructures in order to avoid error propagation.
Neural Attention and Memory Model One of the earliest work with a memory component applied to language processing is memory networks (Weston et al., 2015; Sukhbaatar et al., 2015), which encode facts into vectors and store them in the memory for question answering (QA). Following their success, Xiong et al. (2016) proposed dynamic memory networks (DMN) to additionally capture position and temporality of transitive reasoning steps for different QA tasks. The idea is to encode important knowledge and store it into memory for future usage with attention mechanisms. Attention mechanisms allow neural network models to selectively pay attention to specific parts. There are also various tasks showing the effectiveness of attention mechanisms.
However, most previous work focused on the classification or prediction tasks (predicting a single word given a question), and there are few studies for NLU tasks (slot tagging). Based on the
fact that the linguistic or knowledge-based substructures can be treated as prior knowledge to benefit language understanding, this work borrows the idea from memory models to improve NLU. Unlike the prior NLU work that utilized representations learned from knowledge bases to enrich features of the current sentence, this paper directly learns a sentence representation incorporating memorized substructures with an automatically decided attention mechanism in an end-toend manner.
3 Knowledge-Guided Structural Attention Networks (K-SAN)
Given an utterance with a sequence of words/tokens ~s = w1, ..., wT , our NLU model is to predict corresponding semantic tags ~y = y1, ..., yT for each word/token by incorporating knowledge-guided structures. The proposed model is illustrated in Figure 2. The knowledge encoding module first leverages external knowledge to generate a linguistic structure for the utterance, where a discrete set of knowledge-guided substructures {xi} is encoded into a set of vector representations (§ 3.1). The model learns the representation for the whole sentence by paying different attention on the substructures (§ 3.2). Then the learned vector encoding the knowledge-guided structure is used for improving the semantic tagger (§ 4).
3.1 Knowledge Encoding Module
The prior knowledge obtained from external resources, such as dependency relations, knowledge bases, etc., provides richer information to help decide the semantic tags given an input utterance. This paper takes dependency relations as an example for knowledge encoding, and other structured relations can be applied in the same way. The input utterance is parsed by a dependency parser, and the substructures are built according to the paths from the root to all leaves (Chen and Manning, 2014). For example, the dependency parsing of the utterance “show me the flights from seattle to san francisco” is shown in Figure 3, where the associated substructures are obtained from the parsing tree for knowledge encoding. Here we do not utilize the dependency relation labels in the experiments for better generalization, because the labels may not be always available for different knowledge resources. Note that the number of substruc-
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
knowledge-guided structure {xi}
Knowledge Encoder
Sentence Encoder
Inner Product
u
mi
Knowledge Attention Distributionpi
Encoded Knowledge Representation
Weighted Sum
∑
h
o
Knowledge-Guided Representation
slot tagging sequence
s
y
show me the flights from seattle to san francisco
ROOT
Input Sentence
ht-1 ht+1ht W W W W
wt-1
yt-1
U
Mwt
U
wt+1
U
V
yt
V
yt+1
V
MM
RNN Tagger
Knowledge Encoding Module
Mkg
Min Mout
“show me the flights from seattle to san francisco”
Figure 2: The illustration of knowledge-guided structural attention networks (K-SAN) for NLU.
show
me
the
flights
from
seattle
to
san
francisco
ROOT
1.
3.
4.
2. 1. show me
2. show flights the 3. show flights from seattle 4. show flights to francisco san
Sentence s
show me the flights from seattle to san francisco
Knowledge-Guided Substructure xi
Figure 3: The knowledge-guided substructures of dependency parsing on an example sentence.
tures may be less than the number of words in the utterance, because non-leaf nodes do not have corresponding substructure in order to reduce the duplicated information in the model. The top-left component of Figure 2 illustrates the module for modeling knowledge-guided substructures.
3.2 Model Architecture
The model embeds all knowledge-guided substructures into a continuous space and stores embeddings of all x’s in the knowledge memory. The representation of the input utterance is then compared with encoded knowledge representations to integrate the carried structure guided by knowledge via an attention mechanism. Then the knowledge-guided representation of the sentence is taken together with the word sequence for estimating the semantic tags. Four main procedures are described below.
Encoded Knowledge Representation To store the knowledge-guided structure, we convert each substructure (e.g. path starting from the root to the leaf in the dependency tree), xi, into a structure vector mi with dimension d by embedding
the substructure in a continuous space through the knowledge encoding network Mkg. The input utterance s is also embedded to a vector u with the same dimension through the network model Min. We apply the three types for knowledge encoding models, Mkg and Min, in order to model multiple words from a substructure xi or an input sentence s into a vector representation: 1) fully-connected neural networks (NN) with linear activation, 2) recurrent neural networks (RNN), and 3) convolutional neural networks (CNN) with a window size 3 and a max-pooling operation. For example, one of substructures shown in Figure 3, “show flights seattle from”, is encoded into a vector embedding. In the experiments, the weights of Mkg and Min are tied together based on their consistent ability of sequence encoding.
Knowledge Attention Distribution In the embedding space, we compute the match between the current utterance vector u and its substructure vector mi by taking their inner product followed by a softmax.
pi = softmax(uTmi), (1)
where pi can be viewed as attention distribution for modeling important substructures from external knowledge in order to understand the current utterance.
Sentence Representation In order to encode the knowledge-guided structure, a vector h is a sum over the encoded knowledge embeddings weighted by the attention distribution.
h = ∑ i pimi, (2)
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
which indicates that the sentence pays different attention to different substructures guided from external knowledge. Because the function from input to output is smooth, we can easily compute gradients and back propagate through it. Then the sum of the substructure vector h and the current input embedding u are then passed through a neural network model Mout to generate an output knowledge-guided representation o,
o =Mout(h+ u), (3)
where we employ a fully-connected dense network for Mout.
Sequence Tagging To estimate the tag sequence ~y corresponding to an input word sequence ~s, we use an RNN module for training a slot tagger, where the knowledge-guided representation o is fed into the input of the model in order to incorporate the structure information.
~y = RNN(o,~s) (4)
4 Recurrent Neural Network Tagger
4.1 Chain-Based RNN Tagger
Given ~s = w1, ..., wT , the model is to predict ~y = y1, ..., yT where the tag yi is aligned with the word wi. We use the Elman RNN architecture, consisting of an input layer, a hidden layer, and an output layer (Elman, 1990). The input, hidden and output layers consist of a set of neurons representing the input, hidden, and output at each time step t, wt, ht, and yt, respectively.
ht = φ(Wwt + Uht−1), (5)
ŷt = softmax(V ht), (6)
where φ is a smooth bounded function such as tanh, and ŷt is the probability distribution over of semantic tags given the current hidden state ht. The sequence probability can be formulated as
p(~y | ~s) = ∏ i p(yi | w1, ..., wi). (7)
The model can be trained using backpropagation to maximize the conditional likelihood of the training set labels.
To overcome the frequent vanishing gradients issue when modeling long-term dependencies, gated RNN was designed to use a more sophisticated activation function, such as long shortterm memory (LSTM) and gated recurrent unit
(GRU) (Hochreiter and Schmidhuber, 1997; Cho et al., 2014). RNNs employing either of these recurrent units have been shown to perform well in tasks that require capturing long-term dependencies (Mesnil et al., 2015; Yao et al., 2014; Graves et al., 2013; Sutskever et al., 2014). This paper applies RNN with GRU cells to allow each recurrent unit to adaptively capture dependencies of different time scales (Cho et al., 2014; Chung et al., 2014), because RNN-GRU can yield comparable performance as RNN-LSTM with need of fewer parameters and less data for generalization (Chung et al., 2014; Jozefowicz et al., 2015).
4.2 Knowledge-Guided RNN Tagger
In order to model the encoded knowledge from previous turns, for each time step t, the knowledge-guided sentence representation o in (3) is fed into the RNN model together with the word wt. For the plain RNN, the hidden layer can be formulated as
ht = φ(Mo+Wwt + Uht−1) (8)
to replace (5) as illustrated in the right block of Figure 2. RNN-GRU can incorporate the encoded knowledge in the similar way, where Mo can be added into gating mechanisms for modeling contextual knowledge similarly.
4.3 Joint RNN Tagger
Because the chain-based tagger and the knowledge-guided tagger carry different information, the joint RNN tagger is proposed to balance the information between two model architectures. Figure 4 presents the architecture of
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
the joint RNN tagger.
h1t = φ(W 1wt + U 1ht−1), (9) h2t = φ(Mo+W 2wt + U 2ht−1), (10)
ŷt = softmax(V (αh1t + (1− α)h2t )),(11)
where α is the weight for balancing chainbased and knowledge-guided information. By jointly considering chain-based information (h1t ) and knowledge-guided information (h2t ), the joint RNN tagger is expected to achieve better generalization, and the performance may be less sensitive to poor structures from external knowledge. In the experiments, α is set to 0.5 for balancing two sides. The objective of the proposed model is to maximize the sequence probability p(~y | ~s) in (7), and the model can be trained in an endto-end manner, where the error would be backpropagated through the whole architecture.
5 Experiments
5.1 Experimental Setup
The dataset for experiments is the benchmark ATIS corpus, which is extensively used by the NLU community (Mesnil et al., 2015). There are 4978 training utterances selected from Class A (context independent) in the ATIS-2 and ATIS-3, while there are 893 utterances selected from the ATIS-3 Nov93 and Dec94. In the experiments, we only use lexical features. In order to show the robustness to data scarcity, we conduct the experiments with 3 different sizes of training data (Small, Medium, and Large). The evaluation metrics for NLU is F-measure on the predicted slots.
For experiments with K-SAN, we parse all data with the Stanford dependency parser (Chen and Manning, 2014) and represent words as their embeddings trained on the in-domain data, where the parser is pre-trained on PTB. The loss function is cross-entropy, and the optimizer we use is adam with the default setting (Kingma and Ba, 2014), where the learning rate λ = 0.001, β1 = 0.9, β2 = 0.999, and = 1e−08. The maximum iteration for training our K-SAN models is set as 300. The dimensionality of input word embeddings is 100, and the hidden layer sizes are in {50, 100, 150}. The dropout rates are set as {0.25, 0.50}. All reported results are from the joint RNN tagger, and the hyperparameters are tuned in the dev set for all experiments.
5.2 Baseline
To validate the effectiveness of the proposed model, we compare the performance with the following baselines. • Baseline:
– CRF Tagger (Tur et al., 2010): predicts a semantic slot for each word with a context window (size = 5). – RNN Tagger (Mesnil et al., 2015): predicts a semantic slot for each word. – CNN Encoder-Tagger (Kim, 2014): tag semantic slots with consideration of sentence embeddings learned by a convolutional model.
• Structural: The NLU models utilize linguistic information when tagging slots, where DCNN and Tree-RNN are the state-of-theart approaches for embedding sentences with linguistic structures.
– CRF Tagger (Tur et al., 2010): predicts slots based on the lexical (5-word window) and syntactic (dependent head in the parsing tree) features. – DCNN (Ma et al., 2015a): predicts slots by incorporating sentence embeddings learned by a convolutional model with consideration of dependency tree structures. – Tree-RNN (Tai et al., 2015): predicts slots with sentence embeddings learned by an RNN model based on the tree structures of sentences.
5.3 Slot Filling Results
Table 1 shows the performance of slot filling on different size of training data, where there are three datasets (Small, Medium, and Large use 1/40, 1/10, and whole training data). For baselines (models without knowledge features), CNN Encoder-Tagger achieves the best performance on all datasets.
Among structural models (models with knowledge encoding), Tree-RNN Encoder-Tagger performs better for Small data but slightly worse than the DCNN Encoder-Tagger.
CNN (Kim, 2014) performs better compared to DCNN (Ma et al., 2015a) and Tree-RNN (Tai et al., 2015), even though CNN does not leverage external knowledge when encoding sentences. When comparing the NLU performance between baselines and other state-of-the-art structural mod-
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Model Dataset Encoder (Mkg/Min) Knowledge Tagger Small Medium Large
Baseline - 7 CRF 58.94 78.74 89.73 - 7 RNN 68.58 84.55 92.97 CNN 7 RNN 73.57 85.52 93.88 Structural - 3 CRF 59.55 78.71 90.13 DCNN 3 RNN 70.24 83.80 93.25 Tree-RNN 3 RNN 73.50 83.92 92.28 Proposed K-SAN (NN) 3 RNN 74.11† 85.97 93.98†
K-SAN (RNN) 3 RNN 73.13 86.85† 94.97† K-SAN (CNN) 3 RNN 74.60† 87.99† 94.86†
Table 1: The F1 scores of predicted slots on the different size of ATIS training examples, where K-SAN utilizes the dependency relations parsed from the Stanford parser. Small: 1/40 set; Medium: 1/10 set; Large: original set. († indicates that the performance is significantly better than all baseline models with p < 0.05 in the t-test.)
find nonstop flights from salt lake city to new york on saturday april ninth
find nonstop flights from salt lake city to new york on saturday april ninth
find nonstop flights from salt lake city to new york on saturday april ninth
flight_stop fromloc.city_name toloc.city_name depart_date. day_name depart_date. month_name depart_date. day_number
Small
Medium
Large
Figure 5: The visualization of the decoded knowledge-guided structural attention for both relations and words learned from different size of training data. Relations and words with darker color indicate higher attention weights generated by the proposed K-SAN with CNN. The slot tags are shown in the figure for reference. Note that the dependency relations are incorrectly parsed by the Stanford parser in this example, but our model is still able to benefit from the structural information.
els, there is no significant difference. This suggests that encoding sentence information without distinguishing substructure may not capture salient semantics in order to improve understanding performance.
Among the proposed K-SAN models, CNN for encoding performs best on Small (75% on F1) and Medium (88% on F1), and RNN for encoding performs best on the Large set (95% on F1). Also, most of the proposed models outperform all baselines, where the improvement for the small dataset is more significant. This suggests that the proposed models carry better generalization and are less sensitive to unseen data. For example, given an utterance “which flights leave on monday from montreal and arrive in chicago in the morning”, “morning” can be correctly tagged with a semantic tag B-arrive time.period of day by K-SAN, but it is incorrectly tagged with Bdepart time.period of day by baselines, because knowledge guides the model to pay correct atten-
tion to salient substructures. The proposed model presents the state-of-the-art performance on the large dataset, showing the effectiveness of leveraging knowledge-guided structures for learning embeddings that can be used for specific tasks and the robustness to data scarcity and mismatch.
5.4 Attention Analysis
In order to show the effectiveness of boosting performance by learning correct attention from much smaller training data through the proposed model, we present the visualization of the attention for both words and relations decoded by K-SAN with CNN in the Figure 5. The darker color of blocks and lines indicates the higher attention for words and relations respectively. From the figure, the words and the relations with higher attention are the most crucial parts for predicting correct slots, e.g. origin, destination, and time. Furthermore, the difference of attention distribution between three datasets is not significant; this suggests that
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Approach Knowledge (Max #Substructure) Small Medium Large
CRF Syntax: Dependency Tree
Stanford - 59.55 78.71 90.13 SyntaxNet - 61.09 78.87 90.92
Semantics: AMR Graph Rule-Based - 59.55 79.15 89.97 JAMR - 61.12 78.64 90.25
K-SAN (CNN) Syntax: Dependency Tree
Stanford 53 74.60 87.99 94.86 SyntaxNet 25 74.35 88.40 95.00
Semantics: AMR Graph Rule-Based 19 74.32 88.14 94.85 JAMR 8 74.27 88.27 94.89
Table 2: The F1 scores of predicted slots with knowledge from different resources.
Sentence s show me the flights from seattle to san francisco
Knowledge-Guided Substructure xi
1. show you 2. show flight seattle 3. show flight san francisco 4. show i
show
you
flight I
1.
2.
4.
city
city
Seattle
San Francisco 3.
Figure 6: The knowledge-guided substructures of the AMR graph.
our proposed model is able to pay correct attention to important substructures guided by the external knowledge even the training data is scarce.
5.5 Knowledge Generalization
In order to show the capacity of generalization to different knowledge resources, we perform the KSAN model for different knowledge bases. Below we compare two types of knowledge formats: dependency tree and Abstract Meaning Representation (AMR). AMR is a semantic formalism in which the meaning of a sentence is encoded as a rooted, directed, acyclic graph (Banarescu et al., 2013), where nodes represent concepts, and labeled directed edges represent the relations between two concepts. The formalism is based on propositional logic and neo-Davidsonian event representations (Parsons, 1990; Davidson, 1967). The semantic concepts in AMR were leveraged to benefit multiple NLP tasks (Liu et al., 2015). Unlike syntactic information from dependency trees, the AMR graph contains semantic information, which may offer more specific conceptual relations. Figure 6 shows an AMR graph associated with the example utterance as same as one in Figure 3 and how the knowledge-guided substructures are constructed.
Table 2 presents the performance of CRF and K-SAN with CNN taggers that utilize dependency relations and AMR edges as knowledge guidance on the same datasets, where CRF takes the
head words from either dependency trees or AMR graphs as additional features and K-SAN incorporates knowledge-guided substructures. The dependency trees are obtained from the Stanford dependency parser or the SyntaxNet parser1, and AMR graphs are generated by a rule-based AMR parser or JAMR2.
Among four knowledge resources (different types and obtained from different parsers), all results show the similar performance for three sizes of datasets. The maximum number of substructures for the dependency tree is larger than the number in the AMR graph (53 and 25 v.s. 19 and 8), because syntax is more general and may provide richer cues for guiding more attention while semantics is more specific and may offer stronger guidance. In sum, the models applying four different resources achieve similar performance, and all significantly outperform the state-of-the-art NLU tagger, showing the effectiveness, generalization, and robustness of the proposed K-SAN model.
6 Conclusion
This paper proposes a novel model, knowledgeguided structural attention networks (K-SAN), that leverages prior knowledge as guidance to incorporate non-flat topologies and learn suitable attention for different substructures that are salient for specific tasks. The structured information can be captured from small training data, so the model has better generalization and robustness. The experiments show benefits and effectiveness of the proposed model on the language understanding task, where all knowledge-guided substructures captured by different resources help tagging performance, and the state-of-the-art performance is achieved on the ATIS benchmark dataset.
1https://github.com/tensorflow/models/ tree/master/syntaxnet
2https://github.com/jflanigan/jamr
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This paper proposes a neural network architecture that represent structural linguistic knowledge in a memory network for sequence tagging tasks (in particular, slot-filling of the natural language understanding unit in conversation systems). Substructures (e.g. a node in the parse tree) is encoded as a vector (a memory slot) and a weighted sum of the substructure embeddings are fed in a RNN at each time step as additional context for labeling.  -----Strengths-----  I think the main contribution of this paper is a simple way to  flatten  structured information to an array of vectors (the memory), which is then connected to the tagger as additional knowledge. The idea is similar to structured   syntax-based attention (i.e. attention over nodes from treeLSTM); related work includes Zhao et al on textual entailment, Liu et al. on natural language inference, and Eriguchi et al. for machine translation. The proposed substructure encoder is similar to DCNN (Ma et al.): each node is embedded from a sequence of ancestor words. The architecture does not look entirely novel, but I kind of like the simple and practical approach compared to prior work.  -----Weaknesses-----  I'm not very convinced by the empirical results, mostly due to the lack of details of the baselines. Comments below are ranked by decreasing importance.  -  The proposed model has two main parts: sentence embedding and substructure embedding. In Table 1, the baseline models are TreeRNN and DCNN, they are originally used for sentence embedding but one can easily take the node substructure embedding from them too. It's not clear how they are used to compute the two parts.  - The model uses two RNNs: a chain-based one and a knowledge guided one. The only difference in the knowledge-guided RNN is the addition of a  knowledge  vector from the memory in the RNN input (Eqn 5 and 8). It seems completely unnecessary to me to have separate weights for the two RNNs. The only advantage of using two is an increase of model capacity, i.e. more parameters. Furthermore, what are the hyper-parameters   size of the baseline neural networks? They should have comparable numbers of parameters.  - I also think it is reasonable to include a baseline that just input additional knowledge as features to the RNN, e.g. the head of each word, NER results etc.  - Any comments   results on the model's sensitivity to parser errors?  Comments on the model:  - After computing the substructure embeddings, it seems very natural to compute an attention over them at each word. Is there any reason to use a static attention for all words? I guess as it is, the  knowledge  is acting more like a filter to mark important words. Then it is reasonable to include the baseline suggest above, i.e. input additional features.  - Since the weight on a word is computed by inner product of the sentence embedding and the substructure embedding, and the two embeddings are computed by the same RNN CNN, doesn't it means nodes   phrases similar to the whole sentence gets higher weights, i.e. all leaf nodes?  - The paper claims the model generalizes to different knowledge but I think the substructure has to be represented as a sequence of words, e.g. it doesn't seem straightforward for me to use constituent parse as knowledge here.  Finally, I'm hesitating to call it  knowledge . This is misleading as usually it is used to refer to world   external knowledge such as a knowledge base of entities, whereas here it is really just syntax, or arguably semantics if AMR parsing is used.  -----General Discussion-----  This paper proposes a practical model which seems working well on one dataset, but the main ideas are not very novel (see comments in Strengths). I think as an ACL paper there should be more takeaways. More importantly, the experiments are not convincing as it is presented now. Will need some clarification to better judge the results.  -----Post-rebuttal-----  The authors did not address my main concern, which is whether the baselines (e.g. TreeRNN) are used to compute substructure embeddings independent of the sentence embedding and the joint tagger. Another major concern is the use of two separate RNNs which gives the proposed model more parameters than the baselines. Therefore I'm not changing my scores.


Probabilistic Regular Graph Languages
Distributions over strings and trees can be represented by probabilistic regular languages, which characterize many models in natural language processing. Recently, several datasets have become available which represent natural language phenomena as graphs, so it is natural to ask whether there is an equivalent of probabilistic regular languages for graphs. To answer this question, we review three families of graph languages: Hyperedge Replacement Languages (HRL), which can be made probabilistic; Monadic Second Order Languages (MSOL), which support the crucial property of closure under intersection; and Regular Graph Languages (RGL; Courcelle 1991), a subfamily of both HRL and MSOL which inherits these properties, and has not been widely studied or applied to NLP. We prove that RGLs are closed under intersection and provide an efficient parsing algorithm, with runtime linear in the size of the input graph.
1 Introduction
NLP systems for machine translation, summarization, paraphrasing, and other problems often fail to preserve the compositional semantics of sentences and documents because they model language as bags of words, or at best syntactic trees. To preserve semantics, they must model semantics. In pursuit of this goal, several datasets have been produced which pair natural language with compositional semantic representations in the form of directed acyclic graphs (DAGs), including the Abstract Meaning Represenation Bank (AMR; Banarescu et al. 2013), the Prague Czech-English Dependency Treebank (Hajič et al., 2012), Deep-
bank (Flickinger et al., 2012), and the Universal Conceptual Cognitive Annotation (Abend and Rappoport, 2013). To make use of this data, we require probabilistic models of graphs.
Consider how we might use compositional semantic representations in machine translation (Figure 1). We first parse a source sentence to its semantic representation, and then generate a target sentence from this representation. Stated more formally, we first predict a graph G from a source string s, and then predict a target string t from G, giving us a conditional model P(t, G|s), which we can decompose as P(t, G|s) = P(t|G)P(G|s). Jones et al. (2012) observe that this decomposition can be modeled with a pair of probabilistic synchronous grammars over domains of strings and graphs. Given a domain of source strings Ls, a domain of source graphs LG, a domain of target graphs LG′ and a domain of target strings Lt, these grammars define relationsR ⊆ Ls×LG and R′ ⊆ LG′ ×Lt. Translation is then the solution to the following inference problem, given input s:
arg max t∈Lt ∑ {G|(s,G)∈R∧(G,t)∈R′} P(t|G)P(G|s)
In practical settings the sum over G is typically replaced with an arg max. Either way, we must
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
define probability distributions over the graph domains and efficiently compute their intersection.
For NLP problems in which data is in the form of strings and trees, such distributions can be represented by finite automata (Mohri et al., 2008; Allauzen et al., 2014), which are closed under intersection and can be made probabilistic. It is therefore natural to ask whether there is a family of graph languages with similar properties to finite automata. Recent work in NLP has focused primarily on two families of graph languages: hyperedge replacement languages (HRL; Drewes et al. 1997), a context-free graph rewriting formalism that has been studied in an NLP context by several researchers (Chiang et al., 2013; Peng et al., 2015; Bauer and Rambow, 2016); and DAG automata languages, (Kamimura and Slutzki, 1981), studied by Quernheim and Knight (2012). Thomas (1991) showed that the latter are a subfamily of the monadic second order languages (MSOL), which are of special interest to us, since, when restricted to strings or trees, they exactly characterize the regular languages of each (Büchi, 1960; Büchi and Elgot, 1958; Trakhtenbrot, 1961).
The HRL and MSOL families are incomparable: that is, the context-free graph languages do not contain the regular graph languages, as is the case in languages of strings and trees (Courcelle, 1990). So, while each formalism has appealing characteristics, none appear adequate for the problem outlined above: HRLs can be made probabilistic, but they are not closed under intersection; and while DAGAL and MSOL are closed under intersection, it is unclear how to make them probabilistic (Quernheim and Knight, 2012).1
This paper investigates the regular graph languages (RGL; Courcelle 1991), defined as a restricted form of HRL (§2). The restrictions ensure that RGLs are a subfamily of MSOL. Courcelle (1991) defined regular graph grammars as an auxiliary result of a theoretical research question quite different from ours.2 As a consequence, they have
1Semiring-weighted MSOLs have been defined, where weights may be in the tropical semiring (Droste and Gastin, 2005). However, for the weights to define a probability distribution, they must meet the stronger condition that the sum of multiplied weights over all definable objects is one. This does not appear to have been demonstrated for DAGAL, which violate the sufficient conditions that Booth and Thompson (1973) give for probabilistic languages. We suspect that there are DAGAL (hence MSOL) for which it is not possible.
2The primary research question of Courcelle (1991) is a conjecture which has only quite recently been proven (Bojanczyk and Pilipczuk, 2016).
HRL∗ MSOL† Graphs
CFTL∗
CFL∗
RTL†∗ Trees
RL†∗ Strings
RGL†∗ DAGAL†
Figure 2: Containment relationships for families of regular and context-free string and tree languages, hyperedge replacement languages (HRL), monadic second order definable graph languages (MSOL), directed acyclic graph automata languages (DAGAL), and regular graph languages (RGL). ∗ indicates that the family of languages is probabilistic and † indicates that the family of languages is intersectible.
not been widely studied, and they have never been applied to NLP. We present two new results.
1. We prove that RGLs are closed under intersection (§3).
2. We give a parsing algorithm for RGL that is linear in the size of the input graph (§4).
Figure 2 summarizes the relationship of RGL to other formalisms and their properties. We conclude with a discussion of how RGL relates to other recently-discovered formalisms and its capacity to represent semantics as graphs (§5).
2 Regular Graph Languages
We use the following notation. If n is an integer, [n] denotes the set {1, . . . , n}. IfA is a set, s ∈ A∗ denotes that s is a sequence of arbitrary length, each element of which is in A. We denote by |s| the length of s. A ranked alphabet is an alphabet A paired with an arity function rank: A→ N. Definition 1. A hypergraph over a ranked alphabet Γ is a tuple G = (VG, EG, attG, labG, extG) where VG is a finite set of nodes; EG is a finite set of edges (distinct from VG); attG : EG → V ∗G maps each edge to a sequence of nodes; labG : EG → Γ maps each edge to a label such that |attG(e)| = rank(labG(e)); and extG is an ordered subset of VG called the external nodes of G.
We assume that both the elements of extG and the elements of attG(e) for each edge e are pairwise distinct. An edge e is attached to its nodes by tentacles, each labeled by an integer indicating the node’s position in attG(e) = (v1, . . . , vk). The
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
tentacle from e to vi will have label i, so the tentacle labels lie in the set [k] where k = rank(e). To express that a node v is attached to the ith tentacle of an edge e then we say vert(e, i) = v. Likewise, the nodes in extG are labeled by their position in extG. In figures, the ith external node will be labeled (i). The rank of an edge e is k if att(e) = (v1, . . . , vk) (or equivalently, rank(lab(e)) = k). The rank of a hypergraph G is the size of extG. Example 1. Hypergraph G in Figure 3 has four nodes (shown as black dots) and three hyperedges labeled a, b, and X (shown boxed). The bracketed numbers (1) and (2) denote its external nodes and the numbers between edges and the nodes are tentacle labels. Call the top node v1 and, proceeding clockwise, call the other nodes v2, v3, and v4. Call its edges e1, e2 and e3. Its definition would state:
attG(e1) = (v1, v2) labG(e1) = a
attG(e2) = (v2, v3) labG(e2) = b
attG(e3) = (v1, v4, v3) labG(e3) = X
extG = (v4, v2).
Definition 2. Let G be a hypergraph with an edge e of rank k and letH be a hypergraph also of rank k disjoint from G. The replacement of e by H is the graph G′ = G[e/H]. Its node set VG′ is VG ∪ VH where the ith node of e in G is fused with the ith external node of VH . Its hyperedge set EG′ is (EG−{e})∪EH . For each e′ ∈ EG′ , attG′(e′) = attG(e′) if e′ ∈ EG and attG′(e′) = attH(e′) if e′ ∈ EH . For each e′ ∈ EG′ , labG′(e′) = labG(e′) if e′ ∈ EG and labG′(e′) = labH(e′) if e′ ∈ EH . Its external node list is ext′G = extG. Example 2. Replacement is shown in Figure 3. We denote the replacement as G[X/H] since the edge is unambiguous given its label.
2.1 Hyperedge Replacement Grammars
Definition 3. A hyperedge replacement grammar G = (NG , TG , PG , SG) consists of a finite set of ranked nonterminal symbols NG , a finite set of ranked terminal symbols TG (disjoint from NG), a finite set of productions PG , and a start symbol SG ∈ NG . Every production in PG is of the form X → G where X ∈ NG and G is a hypergraph over NG ∪ TG . For each production, the rank of e must equal the rank of G.
For each production p : X → G, we will use L(p) to refer to X (the left-hand side of p) and R(p) to refer to G (the right-hand side of p). An
edge is a terminal edge if its label is terminal and a nonterminal edge if its label is nonterminal. A graph is a terminal graph if all of its edges are labeled with terminal symbols. The terminal subgraph of a graph is the subgraph consisting of all terminal edges and their endpoints.
Given a HRG G, we say that graph G derives graph G′, denoted G → G′, iff there is an edge e ∈ EG and a nonterminal X ∈ NG such that labG(e) = X and G′ = G[e/H], where X → H is in PG . We extend the idea of derivation to its transitive closure G →∗ G′. For every X ∈ NG we also use X to denote the connected graph consisting of a single edge e with lab(e)=X and nodes (v1, . . . , vrank(X)) such that attG(e) = (v1, . . . , vrank(X)), and we define the language LX(G) as {G | X →∗ G ∧ G is terminal}. The language of G is then L(G) = LSG (G). We call the family of languages that can produced by any HRG the hyperedge replacement languages (HRL).
We assume that terminal edges are always of rank 1 or 2, and depict them as directed edges where the direction is determined by the tentacle labels: the tentacle labeled 1 attaches to the source of the edge and the tentacle labeled 2 attaches to the target of the edge, if it exists.
Example 3. Table 1 shows a HRG deriving AMR graphs for sentences of the form ‘I need to want to need to want to ... to want to go’. Figure 4 is a graph derived by the grammar. The grammar is somewhat unnatural, a point we will return to (§5).
2.2 Properties of HRGs
A HRG can be made probabilistic just as a CFG can: for each nonterminal symbol, we define a distribution over the right-hand sides of all productions with that symbol as its left-hand side (Booth and Thompson, 1973). HRLs are not closed under intersection, just as the CFLs (which they general-
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Sp : X
(1)
1 go 1
2 I
arg0 Y Zs : (1)
(2)
1
2
1
arg0
arg1
Xq : W
Y
(2)
(1) 1
2
1 1
2
arg1
arg0 Wt :
(1) 1
want
Yr : Z
X
(2)
(1) 1
2
1 1
2
arg1
arg0 Zu :
(1) 1
need
Table 1: Productions of a HRG. The labels p, q, r, s, t, and u label the productions so that we can refer to them in the text. Note that Y can rewrite in two ways, either via production r or s.
(1)
arg1
arg1
arg1
arg1
need
want
need
want
go
I
arg0
arg0
arg0
arg0 arg0
Figure 4: Graph derived by grammar in Table 1.
ize) are not: emptiness of HRLs is decidable, but given two HRLs simulating CFLs, the emptiness of their intersection is undecidable.
2.3 Regular Graph Grammars
A regular graph grammar (RGG; Courcelle 1991) is a restricted form of HRG. To explain the restrictions, we first require some definitions.
Definition 4. Given a graph G, a path in G from a node v to a node v′ is a sequence
(v0, i1, e1, j1, v1)(v1, i2, e2, j2, v2)
. . . (vk−1, ik, ek, jk, vk) (1)
such that v0 = v, vk = v′, vert(er, ir) = vr−1, and vert(er, jr) = vr, for each r ∈ {1, . . . , k}. The length of this path is k.
A path is terminal if every edge in the path has a terminal label. A path is internal if each vi is internal for 1 ≤ i ≤ k−1. Note that the endpoints v0 and vk of an internal path can be external.
Definition 5. A HRG G is a Regular Graph Grammar if each nonterminal in NG has rank at least one and for each p ∈ PG the following hold:
(C1) R(p) has at least one edge. Either it is a single terminal edge, all nodes of which are external, or each of its edges has at least one internal node.
(C2) Every pair of nodes in R(p) is connected by a terminal and internal path.
Example 4. The grammar in Table 1 is an RGG. Regular string and tree grammars are also RGGs (Figures 5 and 6).
X
(1)
Y Z
(1)1 a
1 2
1 1
b
Figure 6: RGGs generalize regular tree grammars.
We call the family of languages generated by RGGs the regular graph languages (RGLs).
3 RGLs are closed under intersection
Monadic second-order languages (MSOLs; Courcelle and Engelfriet 2011) are graph languages defined by statements in monadic second-order (MSO) logic. An MSO formula ϕ defines the graph language L(ϕ) = {G | G |= ϕ}. A full explanation is beyond the scope of this paper but we provide a brief discussion of MSO in the supplementary materials. Courcelle (1991) proves that restrictions C1 and C2 ensure that any RGL is also a MSOL.
MSOL is trivially closed under intersection, since the conjunction of two MSO statements is
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
also an MSO statement: if ϕ1 and ϕ2 are both MSO formulae, thenL(ϕ1)∩L(ϕ2) = L(ϕ1∧ϕ2). However, this does not guarantee that an arbitrary subfamily of MSOL is closed under intersection— it only guarantees that the intersection of two languages in subfamilyG are in MSOL, not necessarily inG itself. Here, we give a sufficient condition for a subfamily G to be closed under intersection.
Proposition 1. Let G be (1) a subclass of HRG, defined as a restriction on the right-hand sides of its productions that does not depend on nonterminal labels; and (2) MSO-definable. Then for any pair of languages L1, L2 ∈ G, the language L1 ∩ L2 is also in G.
Proof. Since both L1 and L2 are both in HRL and MSOL, we can look at them from both perspectives. Let G1 be a HRG deriving L1 and let φ2 be an MSO statement defining L2. Propositions 1.10 and 4.8 in Courcelle (1990) prove that the intersection of a HR language and an MSO language is in HRL, by constructing a HRG which derives all and only those graphs in the intersection of the two languages.3 This HRG has finitely many nonterminals defined by the cross product of the nonterminals of the original HRG and a finite set of ‘states’ of the MSO.4 The productions of the intersection grammar are copies of the original HRG, with different nonterminal labels. Hence we can construct HRG G∩ such that L(G∩) = L1 ∩ L2 and the productions in G∩ satisfy any restriction that G1 satisfied since the restriction is in terms of the right-hand sides of the productions but not the nonterminal labels. Therefore, G∩ is in G.
RGG satisfies the conditions of Proposition 1, so RGLs are closed under intersection. Importantly, both proofs—that RGL is in MSOL and that RGL is closed under intersection—are constructive, implying that it is possible to construct the intersection grammar.
4 RGL Parsing
To parse RGG, we will exploit the property that every nonterminal including the start symbol has rank at least one (Definition 5), and we assume
3This is a generalization to graphs of the proof that the intersection of a context-free and regular string language is a context-free string language (Bar-Hillel et al., 1961).
4For each MSO-definable language L in some set of all possible graphs L′, there exists a set A, a homomorphism h : L′ → A, and a finite subset C of A such that L = h−1(C). The finite set of ‘states’ here is the set C.
that the corresponding external node is identified in the input graph. This mild assumption may be a reasonable for applications like AMR parsing, where grammars could be designed so that the external node is always the unique root. Later we will relax this assumption.
The availability of an identifiable external node suggests a top-down algorithm, and we take inspiration from a top-down parsing algorithm the predictive top-down parsable grammars, another subclass of HRG (Drewes et al., 2015). These grammars, the graph equivalent of LL(1) string grammars, are incomparable to RGG, but the algorithms are related in their use of top-down prediction and in that they both fix an order of the edges in the right-hand side of each production.
4.1 Top-down Parsing for RGLs
Just as the algorithm of Chiang et al. (2013) generalizes CKY to HRG, our algorithm generalizes Earley’s algorithm (Earley, 1970). Both algorithms operate by recognizing incrementally larger subgraphs of the input graph, using a succinct representation for subgraphs that depends on an arbitrarily chosen marker node m of the input graph.
Definition 6. (Chiang et al. 2013; Definition 6) Let I be a subgraph of G. A boundary node of I is a node which is either an endpoint of an edge in G\I or an external node of G. A boundary edge of I is an edge in I which has a boundary node as an endpoint. The boundary representation of I is the tuple b(I) = 〈bn(I), be(I),m ∈ I〉 where
1. bn(I) is the set of boundary nodes of I 2. be(I) is the set of boundary edges of I 3. (m ∈ I) is a flag indicating whether the
marker node is in I .
Chiang et al. (2013) prove each subgraph has a unique boundary representation, and give algorithms that use only boundary representations to compute the union of two subgraphs, requiring time linear in the number of boundary nodes; and to check disjointness of subgraphs, requiring time linear in the number of boundary edges.
For each production p of the grammar, we impose a fixed order on the edges of R(p), as in Drewes et al. (2015). We discuss this order in detail in §4.2. As in Earley’s algorithm, we use dotted rules to represent partial recognition of productions: X → ē1 . . . ēi−1 · ēi . . . ēn means that we have identified the edges ē1 to ēi−1 and that we must next recognize edge ēi. We write ē and
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
v̄ for edges and nodes in productions and e and v for edges and nodes in a derived graph. When the identity of the sequence is immaterial we abbreviate it as α, for example writing X → ·α.
We present our parser as a deductive proof system (Shieber et al., 1995). The items of the parser are of the form [b(I), p : X → ē1 . . .· ēi . . . ēn, φp] where I is a subgraph that has been recognised as matching ē1, . . . , ēi−1; p : X → ē1, . . . , ēn is a production (called p) in the grammar with the edges in order; and φp : ER(p) → V ∗G is maps the endpoints of edges in R(p) to nodes in G.
For each production, p, we number the nodes in some arbitrary order. Using this, we construct the function φ0p : ER(p) → V ∗R(p) such that for ē ∈ ER(p) if att(ē) = (v̄1, v̄2) then φ0p(ē) = (v̄1, v̄2). As we match edges in the graph with edges in p, we assign the nodes v̄ to nodes in the graph. For example, if we have an edge ē in a production p such that att(ē) = (v̄1, v̄2) and we find an edge e which matches ē, then we update φp to record this fact, written φp[att(ē) = att(e)]. We also use φp to record assignments of external nodes. If we assign the ith external node to v, we write φp[extp(i) = v]. We write φ0p to represent a mapping with no grounded nodes.
Since our algorithm makes top-down predictions based on known external nodes, our boundary representation must cover the case where a subgraph is empty except for these nodes. If at some point we know that our subgraph has external nodes φ(ē), then we use the shorthand φ(ē) rather than the full boundary representation 〈φ(ē), ∅,m ∈ φ(ē)〉.
To keep notation uniform, we use dummy nonterminal S∗ 6∈ NG that derives SG via production p0. For graph G, our system includes the axiom:
[extG, p0 : S∗ →·SG , φ0p0 [extR(p0) = extG]]. Our goal is to prove:
[b(G), S∗ → SG·, φ] where φ is the union of the φps for each production p used to derive G.
As in Earley’s algorithm, we have three inference rules: PREDICT, SCAN and COMPLETE (Table 2). PREDICT is applied when the edge after the dot is nonterminal, assigning any external nodes
that have been identified. SCAN is applied when the edge after the dot is terminal. Using φp, we may already know where some of the endpoints of the edge should be, so it requires the endpoints of the scanned edge to match. COMPLETE requires that a recognized subgraph J match on the external nodes of its edge in the parent graph, and that the combined subgraphs are edge-disjoint.5
Example 5. Using the RGG in Table 1, we show how to parse the graph in Figure 7, which can be derived by applying production s followed by production u, where the external nodes of Y are (v3, v2). Assume the ordering of the edges in production s is arg1, arg0, Z; the top node is v̄1; the bottom node is v̄2; and the node on the right is v̄3; and that the marker node is not in this subgraph— we elide reference to it for simplicity. The external nodes of Y are determined top-down, so the parse of this subgraph is triggered by this item: [(v3, v2), Y →· arg1arg0Z, φs] where φs(arg1) = (v̄1, v3), φs(arg0) = (v̄1, v2), and φs(Z) = (v̄1).
Table 3 shows how we can prove the item
[〈{v3, v2}, {e3, e2}〉, Y → arg1arg0Z·, φ] The boundary representation 〈{v3, v2}, {e3, e2}〉 in this item represents the whole subgraph shown in Figure 7.
v1
v2
v3 . . .
. . .
need (e1)
arg0 (e2)
arg1 (e3)
Figure 7: Top left subgraph of Figure 4. To refer to nodes and edges in the text, they are labeled v1, v2, v3, e1, e2, and e3.
4.2 Normal Ordering
Our algorithm requires a fixed ordering of the edges in the right-hand sides of each production. We will constrain this ordering to exploit the structure of RGG productions, allowing us to bound
5We provide a soundness and completeness proof of the parser in the supplementary materials.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Name Rule Conditions
PREDICT [b(I), p : X → ē1 . . .· ēi . . . ēn, φp][q : Y → α]
[φp(ēi), Y → ·α, φ0q [extR(q) = φp(ēi)]] lab(ēi) = Y
SCAN [b(I), X → ē1 . . .· ēi . . . ēn, φp][e = edglab(ēi)(v1, . . . , vm)]
[b(I ∪ {e}), X → ē1 . . .· ēi+1 . . . ēn, φp[att(ēi) = (v1, . . . , vm)]] φp(ēi)(j) ∈ VG ⇒ φp(ēi)(j) = vert(e, j)
COMPLETE [b(I), p : X → ē1 . . .· ēi . . . ēn, φp][b(J), q : Y → α·, φq] [b(I ∪ J), X → ē1 . . .· ēi+1 . . . ēn, φp ∪ φq] φp(ēi) = φq(extR(q))
lab(ēi) = Y EI ∩ EJ = ∅
Table 2: The inference rules for the top-down parser.
Current Item Reason 1. [(v3, v2), Y → · arg1arg0Z, φs] Axiom 2. [〈{v3, v2, v1}, {e3}〉, Y → arg1· arg0Z, φs[att(arg1) = (v1, v3)]] SCAN: 1. and e3 = edgarg1(v1, v3) 3. [〈{v3, v2, v1}, {e3, e2}〉, Y → arg1arg0·Z, φs[att(arg0) = (v1, v2)]] SCAN: 2. and e2 = edgarg0(v1, v2)] 4. [(v1), Z → · need, φ0u[extR(u) = φs(Z)]] PREDICT: 3. and Z → need 5. [〈{v1}, {e1}〉, Z → need·, φu[att(need) = (v1)]] SCAN: 4. and e1 = edgneed(v1) 6. [〈{v3, v2}, {e3, e2}〉, Y → arg1arg0Z ·, φs ∪ φu] COMPLETE: 3. and 5.
Table 3: The steps of recognising that the subgraph shown in Figure 7 is derived from productions r2 and u in the grammar in Table 1.
(1) X (2)a 1 2 b
Figure 8: This graph cannot be normally ordered.
parsing complexity. If s = ē1 . . . ēn is an order, define si:j = ēi . . . ēj .
Definition 7. Let s be the order of a right-hand side of a production. s is normal if it has the following properties:
1. ē1 is connected to an external node, 2. s1:j is a connected graph for all j ∈ {1, . . . , n},
3. if ēi is nonterminal, each endpoint of ēi must be shared with ēj where ēj is terminal and j < i.
Example 6. The ordering of the edges of production s in Example 5 is normal.
Arbitrary HRGs do not necessarily admit a normal ordering. For example, the graph in Figure 8 cannot satisfy properties 2 and 3 simultaneously. However, RGGs do admit a normal ordering.
Proposition 2. If G is an RGG, for every p ∈ PG , there is a normal ordering of the edges in R(p).
Proof. If R(p) contains a single node then it must be an external node and it must have a terminal edge attached to it since R(p) must contain at least one terminal edge. If R(p) contains multiple nodes then by C2 there must be terminal internal paths between all of them, so there must be a terminal edge attached to the external node, which we use to satisfy property 1. We select terminal edges once one of their endpoints is connected to an ordered edge, and nonterminal edges once all endpoints are connected to ordered edges, possible by C2. Therefore, properties 2 and 3 are satisfied.
Normal ordering tightly constrains recognition of edges. Property 3 ensures that when we apply PREDICT, the external nodes of the predicted edge are all bound to specific nodes in the graph. Properties 1 and 2 ensures that when we apply SCAN, at least one endpoint of the edge is bound.
4.3 Parsing Complexity
Assume a normally-ordered RGG. Let the maximum number of edges in the right-hand side of any
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
production be m; the maximum number of nodes in any right-hand side of a production k; the maximum degree of any node in the input graph d; and the number of nodes in the input graph n.
Remark 1. The maximum number of nodes in any right-hand side of a production (k) is also the maximum number of boundary nodes for any subgraph in the parser.
COMPLETE combines subgraphs I and J only when the entire subgraph derived from Y has been recognized. Boundary nodes of J are also boundary nodes of I since they are nodes in the terminal subgraph of R(p) where Y connects. The boundary nodes of I ∪J are also bounded by k since are a subset of the boundary nodes of I .
Remark 2. Given a boundary node, there are at most (dm)k−1 ways of identifying the remaining boundary nodes of a subgraph that is isomorphic to the terminal subgraph of the right-hand side of a production.
The terminal subgraph of each production is connected by C2, with a maximum path length of m. For each edge in the path, there are at most d subsequent edges. Hence for the k − 1 remaining boundary nodes there are (dm)k−1 ways of choosing them.
We count instantiations of COMPLETE for an upper bound on complexity (McAllester, 2002), using similar logic to (Chiang et al., 2013). The number of boundary nodes of I, J and I ∪ J is at most k. Therefore, if we choose an arbitrary node to be some boundary node of I ∪J , there are at most (dm)k−1 ways of choosing its remaining boundary nodes. For each of these nodes, there are at most (3d)k states of their attached boundary edges: in I , in J , or in neither. The total number of instantiations is thenO(n(dm)k−1(3d)k), linear in the number of input nodes and exponential in the degree of the input graph. In the case of the AMR dataset, the maximum node degree is 17 and the average degree is 2.12.
We observe that RGGs could be relaxed to produce graphs with no external nodes by adding a dummy nonterminal S′ with rank 0 and a single production S′ → S. To adapt the parsing algorithm, we would first need to guess where the graph starts. This would add a factor of n to the complexity as the graph could start at any node, requiring n runs of the algorithm.
5 Discussion and Conclusions
RGG supports probabilistic interpretation and is closed under intersection, making it similar to regular families of string and tree languages that are widely used in NLP. The constraints of RGG also enable more efficient parsing than general HRG, and this tradeoff is reasonable since HRG is very expressive—when generating strings, it can express non-context-free languages (Engelfriet and Heyker, 1991; Bauer and Rambow, 2016), far more power than needed to express semantic graphs. On the other hand, RGG is so constrained that it may not be expressive enough: it would be more natural to derive the graph in Figure 4 from outermost to innermost predicate; but constraint C2 makes it difficult to express this, and the grammar in Table 1 does not. Perhaps we need less expressivity than HRG but more than RGG.
Since RGLs are a subfamily of both HRL and MSOL, they inherit probability and intersection closure, respectively. Courcelle (1991) discusses exactly those languages that are both HRL and MSOL, which he calls strongly contextfree languages (SCFL).6 SCFLs are defined nonconstructively, but it is natural to ask whether other, less restrictive subfamilies of SCFLs can be constructed using similar mathematical tools. Courcelle (1991) identifies the family of seriesparallel graphs as one such family, but it seems of little relevance to NLP. However, there are two recent independently-developed formalisms that may be useful. Tree-like Grammars (TLG; Matheja et al. 2015) and Restricted DAG Grammars (RDG; Björklund et al. 2016), both explicitly defined as restrictions on HRG. TLGs are in MSOL (and are closed under intersection) but we do not yet know if RDG is a subfamily of MSOL, or whether they are closed under intersection. They are both incomparable to RGG, but they share important characteristics, including the fact that the terminal subgraph of every production is connected. This means that our top-down parsing algorithm is applicable to both. In addition, if RDGs are in fact MSOL, Proposition 1 applies to them and means they are closed under intersection. We conjecture that larger, less restrictive subfamilies of SCFLs may be found based on a weaker restriction of connected terminal subgraphs, and we plan to explore these questions in future work.
6Courcelle’s definition of strongly context-free is unrelated to use of this term in NLP.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

The paper is concerned in finding such a family of graph languages that is closed under intersection and can be made probabilistic.  - Strengths:  The introduction shows relevance, the overall aim, high level context and is nice to read. The motivation is clear and interesting.  The paper  is extremely clear but requires close reading and much formal background. It nicely takes into account certain differences in terminology.  It was interesting to see how the hyper-edge grammars generalize familiar grammars  and Earley's algorithm.  For example, Predict applies to nonterminal edges, and Scan applies to terminal edges.    If the parsing vs. validation in NLP context is clarified, the paper is useful because it is formally correct, nice contribution, instructive and can give new ideas to other researchers.    The described algorithm can be used in semantic parsing to rerank hypergraphs that are produced by another parser.   In this restricted way, the method can be part of the machinery what we in NLP use in natural language parsing and thus relevant to the ACL.  - Weaknesses:  Reranking use is not mentioned in the introduction.  It would be a great news in NLP context if an Earley parser would run in linear time for NLP grammars (unlike special kinds of formal language grammars).  Unfortunately, this result involves deep assumptions about the grammar and the kind of input.   Linear complexity of parsing of an input graph seem right for a top-down deterministic grammars but the paper does not recognise the fact that an input string in NLP usually gives rise to an exponential number of graphs.  In other words, the parsing complexity result must be interpreted in the context of graph validation or where one wants to find out a derivation of the graph, for example, for the purposes of graph transduction via synchronous derivations.  To me, the paper should be more clear in this as a random reader may miss the difference between semantic parsing (from strings) and parsing of semantic parses  (the current work).  There does not seem to be any control of the linear order of 0-arity edges.  It might be useful to mention that if the parser is extended to string inputs with the aim to find the (best?) hypergraph for a given external nodes, then the item representations of the subgraphs must also keep track of the covered 0-arity edges.                          This makes the string-parser variant exponential.    - Easily correctable typos or textual problems:  1)  Lines 102-106 is misleading.   While intersection and probs are true,  such distribution  cannot refer to the discussion in the above.  2) line 173:  I think you should rather talk about validation or recognition algorithms than parsing algorithms as  parsing  in NLP means usually completely different thing that is much more challenging due to the lexical and structural ambiguity.  3) lines 195-196 are unclear:  what are the elements of att_G; in what sense they are pairwise distinct.  Compare Example 1 where ext_G and att_G(e_1) are not disjoint sets.  4) l.206.  Move *rank* definition earlier and remove redundancy.  5) l. 267:  rather  immediately derives , perhaps.  6) 279: add  be   7) l. 352:  give an example of a nontrivial internal path.  8) l. 472:   define a subgraph of a hypergraph  9) l. 417, l.418:  since there are two propositions, you may want to tell how they contribute to what is quoted.  10) l. 458:  add  for   Table:                          Axiom:              this is only place where this is introduced as an axiom.                    Link to the text that says it is a trigger.  - General Discussion:  It might be useful to tell about MSOL graph languages and their yields, which are context-free string languages.                            What happens if the grammar is ambiguous and not top-down deterministic?  What if there are exponential number of parses even for the input graph due to lexical ambiguity or some other reasons.  How would the parser behave then?  Wouldn't the given Earley recogniser actually be strictly polynomial to m or k ?  Even a synchronous derivation of semantic graphs can miss some linguistic phenomena where a semantic distinction is expressed by different linguistic means.                    E.g. one language may add an affix to a verb when another language may express the same distinction by changing the object.  I am suggesting that although AMR increases language independence in parses it may have such cross-lingual challenges.  I did not fully understand the role of the marker in subgraphs.  It was elided later and not really used.  l. 509-510:                 I already started to miss the remark of lines 644-647 at this point.  It seems that the normal order is not unique.  Can you confirm this?  It is nice that def 7, cond 1 introduces lexical anchors to predictions.  Compare the anchors in lexicalized grammars.  l. 760.  Are you sure that non-crossing links do not occur when parsing linearized sentences to semantic graphs?  - Significant questions to the Authors:  Linear complexity of parsing of an input graph seem right for a top-down deterministic grammars but the paper does not recognise the fact that an input string in NLP usually gives rise to an exponential number of graphs.  In other words, the parsing complexity result must be interpreted in the context of graph validation or where one wants to find out a derivation of the graph, for example, for the purposes of graph transduction via synchronous derivations.  What would you say about parsing complexity in the case the RGG is a non-deterministic, possibly ambiguous regular tree grammar, but one is interested to use it to assign trees to frontier strings like a context-free grammar?  Can one adapt the given Earley algorithm to this purpose (by guessing internal nodes and their edges)? Although this question might seem like a confusion, it is relevant in the NLP context.  What prevents the RGGs to generate hypergraphs whose 0-arity edges (~words) are then linearised?   What principle determines how they are linearised?                  Is the linear order determined by the Earley paths (and normal order used in productions) or can one consider an actual word order in strings of a natural language?   There is no clear connection to (non)context-free string languages or sets of (non)projective dependency graphs used in semantic parsing.  What is written on lines 757-758 is just misleading:  Lines 757-758 mention that HRGs can be used to generate non-context-free languages.  Are these graph languages or string languages?    How an NLP expert should interpret the (implicit) fact that RGGs generate only context-free languages?  Does this mean that the graphs are noncrossing graphs in the sense of Kuhlmann & Jonsson (2015)?