Leveraging Behavioral and Social Information for Weakly Supervised Collective Classification of Political Discourse on Twitter
Framing is a political strategy in which politicians carefully word their statements in order to control public perception of issues. Previous works exploring political framing typically analyze frame usage in longer texts, such as Congressional speeches. We present a collection of weakly supervised models which harness collective classification to predict the frames used in political discourse on the microblogging platform, Twitter. Our global probabilistic models show that by combining both lexical features of tweets and network-based behavioral features of Twitter, we are able to increase the average, unsupervised F1 score by 21.52 points over a lexical baseline alone.
1 Introduction
The importance of understanding political discourse on social media platforms is becoming increasingly clear. In recent U.S. presidential elections, Twitter was widely used by all candidates to promote their agenda, interact with supporters, and attack their opponents. Social interactions on such platforms allow politicians to quickly react to current events and gauge interest in and support for their actions. These dynamic settings both emphasize the importance of constructing automated tools for analyzing this content, but also the difficulty of constructing such tools as the language used to discuss new events and political agendas continuously changes. Consequently, the rich social interactions on Twitter can be leveraged to help support such analysis by providing alternatives to direct supervision.
In this paper we focus on political framing, a very nuanced political discourse analysis task, on
Twitter, a relatively unexplored domain for this task. Framing (Entman, 1993; Chong and Druckman, 2007) is employed by politicians to bias the discussion towards their stance by emphasizing specific aspects of the issue. For example, the debate around increasing the minimum wage can be framed as a quality of life issue or as an economic issue. While the first frame supports increasing minimum wage because it betters workers’ lives, the second frame, by conversely emphasizing the costs involved, opposes the increase. Using framing to analyze political discourse has gathered significant interest over the last few years (Tsur et al., 2015; Card et al., 2015; Baumer et al., 2015) as a way to automatically analyze political discourse in Congressional speeches and political news articles. Our dataset consists of the tweets authored by all members of the U.S. Congress from both parties, dealing with several policy issues (e.g., immigration, ACA, etc.). We annotated these tweets by adapting the annotation guidelines developed by Boydstun et al. for Twitter. More details about the annotation process are provided in Section 3.
Twitter issue framing is a challenging multilabel prediction task. Each tweet can be labeled as using one or more frames, out of 17 possibilities, while only providing 140 characters as input to the classifier. Instead of following a supervised path, our main goal in this paper is to evaluate whether the social and behavioral information available on Twitter is sufficient for constructing a reliable classifier for this task. We approach this task using a weakly supervised collective classification approach which leverages the dependencies between tweet frame predictions based on the interactions between their authors.
We model these dependencies by connecting Twitter users that have social connections or behavioral similarity. Social connections are di-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Figure 1: Collective classification framework models the dependency between tweet frame predictions and social interactions and behavioral similarities between their authors.
rected dependencies that represent the followers of each user and retweeting behavior (i.e., user A retweets user B’s content). Interestingly, these social connections capture the flow of influence within political parties; however, the number of connections that cross party lines is extremely low. Instead we rely on capturing behavioral similarity between users to provide us with this information. We construct a temporal histogram for each politician which captures their Twitter activity over time. Users whose Twitter activity peaks at similar times tend to discuss issues in similar ways, making the comparison between their frames usage easier. Figure 1 shows an example of such a network and the prediction dependencies it forms. In addition to the edges, we also represent each politician’s party affiliation and the frequent phrases (e.g., bigrams or trigrams) used by politicians on Twitter.
We compile these structural dependencies into a graphical model using Probabilistic Soft Logic (PSL), a recently introduced probabilistic modeling framework 1. As described in Section 4, PSL combines these aspects declaratively by specifying high level rules over a relational representation of tweet features. The rules are compiled into a graphical model called a hinge-loss Markov random field (Bach et al., 2013), which is used to make the frame prediction. Instead of direct supervision we take a bootstrapping approach by providing a small seed set of keywords adapted from Boydstun et al., for each frame.
1http://psl.cs.umd.edu
Our experiments show that modeling the social and behavioral connections improves F1 prediction scores in both supervised and unsupervised settings, with double the increase in the latter. We apply our unsupervised model to our entire tweets dataset to analyze framing patterns over time by both party and individual politicians. Our analysis provides insight into the usage of framing for identification of aisle-crossing politicians, i.e., those politicians who vote against their party.
2 Related Work
Issue framing is related to the broader challenges of biased language analysis (Recasens et al., 2013; Choi et al., 2012; Greene and Resnik, 2009) and subjectivity (Wiebe et al., 2004). Several previous works have explored framing in public statements, Congressional speeches, and news articles (Fulgoni et al., 2016; Tsur et al., 2015; Card et al., 2015; Baumer et al., 2015). Our approach builds upon the previous work on frame analysis of Boydstun et al., by adapting and applying their annotation guidelines for Twitter.
In recent years there has been growing interest in analyzing political discourse. Most previous work focuses on opinion mining and stance prediction (Sridhar et al., 2015; Hasan and Ng, 2014; Abu-Jbara et al., 2013; Walker et al., 2012; Abbott et al., 2011; Somasundaran and Wiebe, 2010, 2009). Analyzing political tweets has also attracted considerable interest: a recent SemEval task looked into stance prediction2, and more related to our work, Tan et al. have shown how wording choices can affect message propagation on Twitter. Two recent works look into predicting stance (at user and tweet levels respectively) on Twitter using PSL (Johnson and Goldwasser, 2016; Ebrahimi et al., 2016). Other works focus on identifying and measuring political ideologies (Iyyer et al., 2014; Bamman and Smith, 2015; Sim et al., 2013), policies (Nguyen et al., 2015), and voting patterns (Gerrish and Blei, 2012).
Exploiting social interactions and group structure for prediction has also been explored (Sridhar et al., 2015; Abu-Jbara et al., 2013; West et al., 2014). Works focusing on inferring signed social networks (West et al., 2014), stance classification (Sridhar et al., 2015), social group modeling (Huang et al., 2012), and collective classification using PSL (Bach et al., 2015) are closest
2http://alt.qcri.org/semeval2016/task6/
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
to our approach. Unsupervised and weakly supervised models of Twitter data for several various tasks have been suggested, including: profile (Li et al., 2014b) and life event extraction (Li et al., 2014a), conversation modeling (Ritter et al., 2010), and methods for dealing with the unique language used in microblogging platforms (Eisenstein, 2013).
Predicting political affiliation and other characteristics of Twitter users has been explored (Volkova et al., 2015, 2014; Yano et al.; Conover et al., 2011). Other works have focused on sentiment analysis (Pla and Hurtado, 2014; Bakliwal et al., 2013), predicting ideology (Djemili et al., 2014), automatic polls based on Twitter sentiment and political forecasting using Twitter (Bermingham and Smeaton, 2011; O’Connor et al., 2010; Tumasjan et al., 2010), as well as distant supervision applications (Marchetti-Bowick and Chambers, 2012).
3 Data Collection and Annotation
We collected 184,914 of the most recent tweets of members of the U.S. Congress (both the House of Representatives and Senate). Using an average of ten keywords per issue, we filtered out tweets not related to the following six issues of interest: (1) abortion access, (2) the Affordable Care Act (i.e., ACA or Obamacare), (3) gun rights versus gun control, (4) immigration policies, (5) acts of terrorism, and (6) issues concerning the LGBTQ community. Forty politicians (10 Republicans and Democrats, from both the House and Senate), were chosen randomly for annotation.
Two graduate students were trained to annotate each tweet with a frame using the Policy Frames Codebook developed by Boydstun et al.. Brief descriptions of each frame are given in Table 1. During the annotation process, we found that Boydstun’s Frame 15 (Other) was never used. We therefore dropped it from this study. In its place, we propose 3 Twitter-specific frames: Factual (15), Promotion (16), and Personal Sympathy and Support (17). Due to the compound nature of tweets and the consequent possibility of multiple frames per tweet (also discussed in Card et al.), annotators were allowed to label each tweet with multiple frames when one primary frame was not possible. For all such tweets, annotators repeated the annotation process together to determine if the tweets could be represented by a single frame or required
more. We computed the inter-annotator agreement using Cohen’s Kappa statistic and have an agreement of 73.4%, which can be viewed as proof of the difficulty of frame classification for tweets. Table 2 presents the statistics of our tweets dataset, which will be released for the community’s use.
4 Global Models of Twitter Language and Activity
Due to the dynamic nature of political discourse on Twitter, we design weakly supervised PSL models to require as little supervision as possible. The only sources of supervision our approach requires include: unigrams related to the issues, unigrams adapted from the Boydstun et al. Codebook for frames, and political party of the author of the tweets3. The local models described in this section are data-dependent and used to extract and format information from tweets into input for PSL predicates and rules. Our goal to label each tweet with a frame can be defined in PSL notation as a target predicate: FRAME(T, F), where T represents a tweet, and F represents one of the 17 frames listed in Table 1.
4.1 Global Modeling Using PSL
PSL is a declarative modeling language which can be used to specify weighted, first-order logic rules. These rules are compiled into a hinge-loss Markov random field which defines a probability distribution over possible continuous value assignments to the random variables of the model (Bach et al., 2015) 4. This probability density function is represented as:
P (Y | X) = 1 Z exp
( −
M∑ r=1 λrφr(Y , X)
)
where Z is a normalization constant, λ is the weight vector, and
φr(Y,X) = (max{lr(Y, X), 0})ρr
is the hinge-loss potential specified by a linear function lr. The exponent ρr ∈ 1, 2 is optional. Each potential represents the instantiation of a rule, which takes the following form:
λ1 : P1(x) ∧ P2(x, y)→ P3(y) λ2 : P1(x) ∧ P4(x, y)→ ¬P3(y)
P1, P2, P3, and P4 are predicates (e.g., political party, issue, frame, and presence of n-grams) and x, y are variables. Each rule has a weight λ which reflects that rule’s importance and is learned using
3All information will be released with our dataset at: www.***.***.
4This is the opposite of other probabilistic logical models, e.g. MLNs, in which rules are strictly true or false.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
FRAME NUMBER, FRAME NAME, AND BRIEF DESCRIPTION OF FRAME 1. ECONOMIC: Pertains to the economic impacts of a policy 2. CAPACITY & RESOURCES: Pertains to lack of or availability of resources 3. MORALITY & ETHICS: Motivated by religious doctrine, righteousness, sense of responsibility 4. FAIRNESS & EQUALITY: Of how laws, punishments, resources, etc. are distributed among groups 5. LEGALITY, CONSTITUTIONALITY, & JURISDICTION: Including court cases, restriction and expressions of rights 6. CRIME & PUNISHMENT: Policy violation and consequences 7. SECURITY & DEFENSE: Threats or defenses/preemptive actions to protect against threats 8. HEALTH & SAFETY: Includes care access and effectiveness 9. QUALITY OF LIFE: Effects on individual and community life 10. CULTURAL IDENTITY: Culture’s norms, trends, customs 11. PUBLIC SENTIMENT: Pertains to opinions, polling, and demographics 12. POLITICAL FACTORS & IMPLICATIONS: Efforts, stances, filibusters, lobbying, maneuvering, references to other politicians 13. POLICY DESCRIPTION, PRESCRIPTION, & EVALUATION: Discussion about effectiveness of current or proposed policies 14. EXTERNAL REGULATION AND REPUTATION: Interstate and international relationships of the U.S. 15. FACTUAL: Expresses a pure fact, with no political spin 16. PROMOTION: Promotes another person or the author in some way, e.g. television appearances 17. PERSONAL SYMPATHY & SUPPORT: Expresses sympathy, emotional response, or solidarity with others
Table 1: Frames of Boydstun. The first 14 are Boydstun’s frames and the last 3 are our proposed Twitter-specific frames. Boydstun’s original Frame 15 (Other) is omitted from this study.
Tweets BY PARTY BY ISSUEREPUBLICAN DEMOCRAT ABORTION ACA GUNS IMMIGRATION TERRORISM LGBTQ ENTIRE DATASET 48504 43953 6467 35854 15532 13442 15205 6046 LABELED SUBSET 894 1156 170 564 543 233 446 183
Table 2: Statistics of Collected Tweets.
the Expectation-Maximization algorithm in our unsupervised experiments. Using concrete constants a, b (e.g., tweets and words) which instantiate the variables x, y, model atoms are mapped to continuous [0,1] assignments. More important rules (i.e., those with larger weights) are given preference by the model.
4.2 Language Based Models
Unigrams: Using the guidelines provided in the Policy Frames Codebook (Boydstun et al., 2014), we adapted a list of expected unigrams for each frame. For example, unigrams that should be related to Frame 12 (Political Factors & Implications) include: filibuster, lobby, Democrats, Republicans. We expect that if a tweet and frame contain a matching unigram, then that tweet is likely expressed by that frame. The information that tweet T has expected unigram U of frame F is represented with the PSL predicate: HASUNIGRAMF (T, U). This knowledge is then used as input to PSL Model 1 via the rule: HASUNIGRAMF (T, U) →FRAME(T, F) (shown in line 1 of Table 3).
However, not every tweet will have a unigram that matches those in this list. Under the intuition that at least one unigram in a tweet should be similar to a unigram in the list, we designed the fol-
lowing MaxSim metric to compute the maximum similarity between a word in a tweet and a word from the list of unigrams.
MAXSIM(T, F) = argmax u∈F,w∈T SIMILARITY(W,U)
(1) T is a tweet, W is each word in T, and U is each unigram in the list of expected unigrams (per frame). Similarity is the computed word2vec similarity of each word in the tweet with every unigram in the list of unigrams for each frame. The frame F of the maximum scoring unigram is input to the PSL predicate: MAXSIMF (T, F), which indicates that tweet T has the highest similarity to frame F.
Bigrams and Trigrams: In addition to unigrams, we also explored the effects of political party slogans on frame prediction. Slogans are common catch phrases or sayings that people typically associate with different U.S. political parties. For example, Republicans are known for using “repeal and replace”, a trigram, when they discuss the ACA. Similarly, in the 2016 U.S. presidential election, Secretary Hillary Clinton’s campaign slogan became “Love Trumps Hate”. To visualize slogan usage by parties for different issues, we used the entire tweets dataset, including all unlabeled tweets, to extract the top bigrams and trigrams per party for each issue. The
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
TYPES OF MODELS
MODEL NUMBER BASIS OF MODEL
EXAMPLE OF PSL RULES
LANGUAGE BASED
1 Unigrams HASUNIGRAMF (T, U)→FRAME(T, F) 2 Bigrams HASUNIGRAMF (T, U) ∧PARTYBIGRAMP (T, B)→FRAME(T, F) 3 Trigrams HASUNIGRAMF (T, U) ∧PARTYTRIGRAMP (T, TG)→FRAME(T, F)
BEHAVIOR BASED
4 Temporal Activity SAMETIME(T1, T2) ∧FRAME(T1, F)→FRAME(T2, F) 5 Retweet Patterns RETWEETS(T1, T2) ∧FRAME(T1, F)→FRAME(T2, F) 6 Following Network FOLLOWS(T1, T2) ∧FRAME(T1, F)→FRAME(T2, F)
Table 3: Examples of PSL Model Rules. Each model adds to the rules of the previous model. For brevity we show a subset of the rules and omit full model combinations. The full list of rules per model will be released with our dataset.
0
2000
4000
6000
8000
10000
1 5 9 1 3 1 7 2 1 2 5 2 9 3 3 3 7 4 1 4 5 4 9 5 3 5 7 6 1 6 5 6 9 7 3 7 7 8 1 8 5 8 9 9 3 9 7
Fr eq
u en
cy o
f O
cc u
re n
ce s
Bigrams
Democrat Top 100 Bigrams
Abortion ACA Guns Immigration Terrorism LGBTQ
(a) Democrat Bigrams
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
1 5 9 1 3 1 7 2 1 2 5 2 9 3 3 3 7 4 1 4 5 4 9 5 3 5 7 6 1 6 5 6 9 7 3 7 7 8 1 8 5 8 9 9 3 9 7
Fr eq
u en
cy o
f O
cc u
re n
ce s
Bigrams
Republican Top 100 Bigrams
Abortion ACA Guns Immigration Terrorism LGBTQ
(b) Republican Bigrams
0
500
1000
1500
2000
1 5 9 1 3 1 7 2 1 2 5 2 9 3 3 3 7 4 1 4 5 4 9 5 3 5 7 6 1 6 5 6 9 7 3 7 7 8 1 8 5 8 9 9 3 9 7
Fr eq
u en
cy o
f O
cc u
re n
ce s
Trigrams
Democrat Top 100 Trigrams
Abortion ACA Guns Immigration Terrorism LGBTQ
(c) Democrat Trigrams
0
200
400
600
800
1000
1200
1400
1 5 9 1 3 1 7 2 1 2 5 2 9 3 3 3 7 4 1 4 5 4 9 5 3 5 7 6 1 6 5 6 9 7 3 7 7 8 1 8 5 8 9 9 3 9 7
Fr eq
u en
cy o
f O
cc u
re n
ce s
Trigrams
Republican Top 100 Trigrams
Abortion ACA Guns Immigration Terrorism LGBTQ
(d) Republican Trigrams
Figure 2: Distributions of Bigrams and Trigrams by Party.
histograms in Figure 2 show these distributions for the top 100 bigrams and trigrams. Based on these results, we use the top 20 bigrams (e.g., women’s healthcare and immigration reform) and trigrams (e.g. prevent gun violence) as input to PSL predicates PARTYBIGRAMP (T, B) and PARTYTRIGRAMP (T, TG). These rules represent that tweet T has bigram B or trigram TG from the respective phrase lists of either party (i.e., P represents either Democrat or Republican in the rule instantiation).
4.3 Twitter Behavior Based Models
In addition to language based features of tweets, we also exploit the behavioral features of Twitter including similarities between temporal activity and network relationships.
Temporal Similarity: When an event happens politicians are most likely to tweet about that event within hours of its occurrence. Similarly, most politicians tweet about the event most frequently the day of the event and this frequency decreases over time. We expect that the frames used the day of an event will be similar and change over time. To capture this behavior we use the PSL predicate SAMETIME(T1, T2). This indicates that
tweet T1 occurs around the same time as tweet T25. This information is used in Model 4 via rules such as: SAMETIME(T1, T2) & FRAME(T1, F) →FRAME(T2, F), as shown in line 4 of Table 3.
Network Similarity: Finally, we expect that politicians who share ideologies, and thus are likely to frame issues similarly, will retweet and/or follow each other on Twitter. Due to the compound nature of tweets, retweeting with additional comments can add more frames to the original tweet. Additionally, politicians on Twitter are more likely to follow members of their own party or similar non-political entities than those of the opposing party. To capture this network-based behavior we use two PSL predicates: RETWEETS(T1, T2) and FOLLOWS(T1, T2). These predicates indicate that the content of tweet T1 includes a retweet of tweet T2 and that the author of T1 follows the author of T2 on Twitter, respectively. The last two lines of Table 3 show examples of how network similarity is incorporated into PSL rules.
5 Experiments
Experimental Settings: We provide an analysis of our PSL models under both supervised and unsupervised settings. As shown in Table 4, Model 1 in a supervised setting is similar to the traditional bag-of-words baseline, which we use as our baseline to improve upon. Additionally, Model 6 of the supervised, collective network setting represents the best results we can achieve. In the supervised experiments, we used five-fold cross validation with randomly chosen splits.
We also explore the results of our PSL models in an unsupervised setting because the highly dynamic nature of political discourse on Twitter makes it unrealistic to expect annotated data to
5We conducted experiments with different hour and day limits and found that using a time frame of one hour results in the best accuracy.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
SETTING UNIGRAM BASELINE
COLLECTIVE NETWORK
SUPERVISED 66.02 77.79 UNSUPERVISED 37.14 58.66
Table 4: Baseline and Skyline Weighted Averages. Supervised, unigram model is the lowest score the models can achieve. Supervised, collective network model is the highest.
generalize to future discussions. The only source of supervision comes from the initial unigrams lists as described in Section 4. The labeled tweets are used for evaluation only. As seen in Table 4, we are able to improve the unsupervised model to within an F1 score of 7.36 points of the unigram baseline, and 19.13 points of the best supervised score.
Evaluation Metrics: Since each tweet can have more than one frame, our prediction task is a multilabel classification task. The precision of a multilabel model is the ratio of how many predicted labels are correct:
Precision = 1
T T∑ t=1 |Yt ∩ h(xt)| |h(xt)|
(2)
The recall of this model is the ratio of how many of the actual labels were predicted:
Recall = 1
T T∑ t=1 |Yt ∩ h(xt)| |Yt|
(3)
In both formulas, T is the number of tweets, Yt is the true label for tweet t, xt is a tweet example, and h(xt) are the predicted labels for that tweet. The F1 score is computed as the harmonic mean of the precision and recall.
Analysis of Supervised Experiments: Table 5 shows the results of our supervised experiments. Here we can see that by adding Twitter behavior (beginning with Model 4), our behaviorbased models achieve the best F1 scores across all frames. Model 4 achieves the highest results on two frames, suggesting retweeting and network follower information does not help improve the prediction score for these frames. Model 5 similarly achieves the highest prediction for 5 of the frames, suggesting network follower information cannot further improve the score for these frames. Overall, the Twitter behavior based models are able to outperform language based models alone, including the best performing language
model (Model 3) which combines unigrams, bigrams, and trigrams together to collectively infer the correct frame(s).
Analysis of Unsupervised Experiments: In the unsupervised setting, Model 6, the combination of language and Twitter behavior features achieves the best results on 16 of the 17 issues, as shown in Table 6. There are a few interesting aspects of the unsupervised setting which differ from the supervised setting. Six of the frame predictions do worse in Model 2, which is double that of the supervised version. This is likely due to the presence of overlapping bigrams across frames and issues, e.g., “women’s healthcare” could appear in both Frames 4 and 8 and the issues of ACA and abortion. However, all six are able to improve with the addition of trigrams (Model 3), whereas only 1 of 3 frames improves in the supervised setting. This suggests that bigrams may not be as useful as trigrams in an unsupervised setting. Finally, in Model 5, which adds retweet behaviors, we notice that 5 of the frames decrease in F1 score and 11 of the frames have the same score as the previous model. These results suggest that retweet behaviors are not as useful as the follower network relationships in an unsupervised setting.
6 Qualitative Analysis
To explore the usefulness of frame identification in political discourse analysis, we apply our best performing model (Model 6) on the unlabeled dataset to determine framing patterns over time, both by party and individual. Figure 3 shows the results of our frame analysis over time for two issues: ACA and terrorism6. We compiled the predicted frames for tweets from 2014 to 2016 for each party. Figure 4 presents the results of frame prediction for 2015 tweets of individuals for these two issues.
Party Frames: From Figure 3(a) we can see that Democrats mainly use Frames 1, 4, 8, 9, and 15 to discuss ACA, while Figure 3(c) shows that Republicans predominantly use Frames 1, 8, 9, 12, and 13. Though the parties use similar frames, they are used to express different agendas. For example, Democrats use Frame 8 to indicate the positive effect that the ACA has had in granting more Americans health care access. Republicans,
6Due to space, we omit the other 4 issues. These 2 were chosen because they are among the most frequently discussed topics in our dataset.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Frame Number Frame RESULTS OF SUPERVISED PSL MODEL FRAME PREDICTIONSMODEL 1 MODEL 2 MODEL 3 MODEL 4 MODEL 5 MODEL 6 1 ECONOMIC 85.19 85.19 86.73 87.72 87.72 89.88 2 CAPACITY & RESOURCES 55.38 61.54 76.71 77.11 77.11 79.55 3 MORALITY 73.39 80.52 86.95 87.5 87.43 87.43 4 FAIRNESS 63.56 67.83 65.19 69.91 79.53 82.35 5 LEGALITY 80.41 80.78 80.79 83.33 81.79 82.16 6 CRIME 54.55 54.55 66.67 76.92 76.92 76.92 7 SECURITY 84.40 82.14 84.10 86.67 86.67 88.48 8 HEALTH 73.50 75.76 75.59 77.46 79.71 79.71 9 QUALITY OF LIFE 69.39 68.00 69.39 72.34 72.34 82.93 10 CULTURAL 75.86 78.57 81.25 81.25 81.25 85.71 11 PUBLIC SENTIMENT 12.25 15.25 24.62 24.24 26.24 29.41 12 POLITICAL 54.21 63.31 74.33 74.42 74.52 74.52 13 POLICY 55.75 58.87 60.25 61.54 64.06 65.06 14 EXTERNAL REGULATION 60.71 59.15 64.71 74.35 74.35 85.71 15 FACTUAL 66.56 68.00 71.43 81.82 80.82 82.85 16 PROMOTION 85.71 86.46 86.58 87.34 87.33 91.76 17 PERSONAL 71.79 71.71 74.73 75.00 77.55 77.55
WEIGHTED AVERAGE 66.02 68.78 72.49 74.40 75.71 77.79
Table 5: F1 Scores of Supervised PSL Models. The highest prediction per frame is marked in bold.
Frame Number Frame RESULTS OF UNSUPERVISED PSL MODEL FRAME PREDICTIONSMODEL 1 MODEL 2 MODEL 3 MODEL 4 MODEL 5 MODEL 6 1 ECONOMIC 31.82 31.52 69.57 72.22 72.22 73.23 2 CAPACITY & RESOURCES 23.38 28.51 40.00 41.18 41.18 41.18 3 MORALITY 28.63 29.41 47.67 53.98 43.06 53.99 4 FAIRNESS 33.49 47.19 59.15 63.50 63.50 64.74 5 LEGALITY 44.58 46.93 58.02 60.64 60.63 64.54 6 CRIME 7.89 7.62 73.33 75.00 75.00 76.92 7 SECURITY 42.50 40.24 51.83 62.09 61.68 64.09 8 HEALTH 48.36 48.79 79.43 86.49 86.49 86.67 9 QUALITY OF LIFE 17.82 21.99 48.89 52.63 52.63 54.35 10 CULTURAL 15.38 15.67 51.22 52.63 52.63 55.56 11 PUBLIC SENTIMENT 15.22 15.72 50.79 53.97 41.03 54.69 12 POLITICAL 49.06 48.20 50.29 46.99 46.99 47.23 13 POLICY 39.88 39.39 37.02 42.77 42.77 43.79 14 EXTERNAL REGULATION 12.66 14.22 44.44 66.67 66.67 71.43 15 FACTUAL 24.64 19.21 70.95 70.37 70.41 78.95 16 PROMOTION 40.11 46.41 48.16 50.96 50.96 52.89 17 PERSONAL 45.36 46.15 59.66 62.99 62.13 71.20
WEIGHTED AVERAGE 37.14 38.79 53.13 56.49 55.54 58.66
Table 6: F1 Scores of Unsupervised PSL Models. The highest prediction per frame is marked in bold.
0
500
1000
1500
2000
2500
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
N u
m b
er o
f Tw
ee ts
Frames
Democrat ACA Tweets
2014 2015 2016
(a) Democrat ACA Frames
0
100
200
300
400
500
600
700
800
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
N u
m b
er o
f Tw
ee ts
Frames
Democrat Terrorism Tweets
2014 2015 2016
(b) Dem. Terrorism Frames
0
200
400
600
800
1000
1200
1400
1600
1800
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
N u
m b
er o
f Tw
ee ts
Frames
Republican ACA Tweets
2014 2015 2016
(c) Republican ACA Frames
0
200
400
600
800
1000
1200
1400
1600
1800
2000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
N u
m b
er o
f Tw
ee ts
Frames
Republican Terrorism Tweets
2014 2015 2016
(d) Rep. Terrorism Frames
Figure 3: Predicted Frames for Tweets from 2014 to 2016 by Party for ACA and Terrorism.
however, use Frame 8 (and Frame 13) to indicate their party’s agenda to replace the ACA with access to different options for health care. Additionally, Democrats use the Fairness & Equality Frame (Frame 4) to convey that the ACA gives minority groups a better chance at accessing health care. They also use Frame 15 to express statistics about enrollment of Americans under the ACA. Finally, Republicans use Frames 12 and 13 to bring attention to their own party’s actions to “repeal and replace” the ACA with different policies.
Figures 3(b) and 3(d) show the party-based framing patterns over time for terrorism related tweets. For this issue both parties use similar frames: 3, 7, 10, 14, 16, and 17, but to express different views. For example, Democrats use Frame
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
0
10
20
30
40
50
60
70
80
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
N u
m b
er o
f Tw
ee ts
Frames
ACA Vote Aisle-Crossing Republicans
Dold Buck Meadows Walker Salmon Poliquin Hanna Jones
(a) Aisle-Crossing Republicans on ACA Votes.
0
5
10
15
20
25
30
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
N u
m b
er o
f Tw
ee ts
Frames
Terrorism Vote Aisle-Crossing Democrats
Clyburn Carson Lee Watson Coleman Cleaver Moore McDermott Lewis Fudge Kaptur Richmond
(b) Aisle-Crossing Democrats on Terrorism Votes.
Figure 4: Predicted Frames for Tweets of Aisle-Crossing Politicians in 2015.
3 to indicate moral responsibility to fight ISIS. Republicans use Frame 3 to frame terrorists or their attacks as a result of “radical Islam”. An interesting pattern to note is seen in Frames 10 and 14 for both parties. In 2015 there is a large increase in the usage of this frame. This seems to indicate that parties possibly adopt new frames simultaneously or in response to the opposing party, perhaps in an effort to be in control of the way the message is delivered through that frame.
Individual Frames: In addition to entire party analysis, we were interested in seeing if frames could shed light on the behavior of aisle-crossing politicians. These are politicians who do not vote the same as the majority vote of their party (i.e., they vote the same as the opposing party). Identifying such politicians can be useful in governments which are heavily split by party, i.e., governments such as the recent U.S. Congress (2015 to 2017), where politicians tend to vote the same as the rest of their party members. For this analysis, we collected five 2015 votes from the House of Representatives on both issues and compiled a list of the politicians who voted opposite to their party.
The most important descriptor we noticed was that all aisle-crossing politicians tweet less frequently on the issue than their fellow party members. This is true for both parties. This behavior could indicate lack of desire to draw attention to one’s stance on the particular issue.
Figure 4(a) shows the framing patterns of aislecrossing Republicans on ACA votes from 2015. Recall from Figure 3 that Democrats mostly use Frames 1, 4, 8, 9, and 15, while Republicans mainly use Frames 1, 8, and 9. In this example, these Republicans are considered aislecrossing votes because they have voted the same as Democrats on this issue. The most interesting pattern to note here is that these Republicans use the same framing patterns as the Republicans (Frames 1, 8, and 9), but they also use the frames that are unique to Democrats: Frames 4 and 15. These latter two frames appear significantly less in the Republican tweets of our entire dataset as well. These results suggest that to predict aisle-crossing Republicans it would be useful to check for usage of typically Democrat-associated frames, especially if those frames are infrequently used by Republicans.
Figure 4(b) shows the predicted frames for aisle-crossing Democrats on terrorism-related votes. We see here that there are very few tweets from these Democrats on this issue and that overall they use the same framing patterns as seen previously: Frames 3, 7, 10, 14, 16, and 17. However, given the small scale of these tweets, we can also consider Frames 12 and 13 to show peaks for this example. This suggests that for aisle-crossing Democrats the use of additional frames not seen in their party might indicate potentially different voting behaviors.
7 Conclusion
In this paper we present the task of collective classification of Twitter data for framing prediction. We show that by incorporating Twitter behaviors such as similar activity times and similar networks, we can increase F1 score prediction. We provide an analysis of our approach in both supervised and unsupervised settings, as well as a real world analysis of framing patterns over time. Our global PSL models can be applied to other domains, such as politics in other countries, simply by changing the initial unigram keywords to reflect the politics of those countries.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths: 1) an interesting task, 2) the paper is very clearly written, easy to follow, 3) the created data set may be useful for other researchers, 4) a detailed analysis of the performance of the model.  - Weaknesses: 1) no method adapted from related work for a result comparison 2) some explanations about the uniqueness of the task and discussion on limitations of previous research for solving this problem can be added to emphasize the research contributions further.   - General Discussion: The paper presents supervised and weakly supervised models for frame classification in tweets. Predicate rules are generated exploiting language-based and Twitter behavior-based signals, which are then supplied to the probabilistic soft logic framework to build classification models. 17 political frames are classified in tweets in a multi-label classification task. The experimental results demonstrate the benefit of the predicates created using the behavior-based signals. Please find my more specific comments below:  The paper should have a discussion on how frame classification differs from stance classification. Are they both under the same umbrella but with different levels of granularity?  The paper will benefit from adding a brief discussion on how exactly the transition from long congressional speech to short tweets adds to the challenges of the task. For example, does past research rely on any specific cross-sentential features that do not apply to tweets? Consider adapting the method of a frame classification work on congressional speech (or a stance classification work on any text) to the extent possible due to its limitations on Twitter data, to compare with the results of this work.  It seems  weakly supervised d and  unsupervised d   these two terms have been interchangeably used in the paper (if this is not the case, please clarify in author response). I believe  weakly supervised  is the more technically correct terminology under the setup of this work that should be used consistently throughout. The initial unlabeled data may not have been labeled by human annotators, but the classification does use weak or noisy labels of some sort, and the keywords do come from experts. The presented method does not use completely unsupervised data as traditional unsupervised methods such as clustering, topic models or word embeddings would.    The calculated Kappa may not be a straightforward reflection of the difficulty of frame classification for tweets (lines: 252-253), viewing it as a proof is a rather strong claim. The Kappa here merely represents the annotation difficulty disagreement. Many factors can contribute to a low value  such as poorly written annotation guidelines, selection of a biased annotator, lack of annotator training etc. (on top of any difficulty of frame classification for tweets by human annotators, which the authors actually intend to relate to). 73.4% Cohen  Kappa is strong enough for this task, in my opinion, to rely on the annotated labels.   Eq (1) (lines: 375-377) will ignore any contextual information (such as negation or conditional hypothetical statements impacting the contributing word) when calculating similarity of a frame and a tweet. Will this have any effect on the frame prediction model? Did the authors consider using models that can determine similarity with larger text units such as perhaps using skip thought vectors or vector compositionality methods?    An ideal set up would exclude the annotated data from calculating statistics used to select the top N bi tri-grams (line: 397 mentions entire tweets data set has been used), otherwise statistics from any test fold (or labeled data in the weakly supervised setup) still leaks into the selection process. I do not think this would have made any difference in the current selection of the bi tri-grams or results as the size of the unlabeled data is much larger, but would have constituted a cleaner experimental setup.    Please add precision and recall results in Table 4.   Minor: please double check any rules for footnote placements concerning placement before or after the punctuation.


Learning a Neural Semantic Parser from User Feedback
We present an approach to rapidly and easily build natural language interfaces to databases for new domains, whose performance improves over time based on user feedback, and requires minimal intervention. To achieve this, we adapt neural sequence models to map utterances directly to SQL with its full expressivity, bypassing any intermediate meaning representations. These models are immediately deployed online to solicit feedback from real users to flag incorrect queries. Finally, the popularity of SQL facilitates gathering annotations for incorrect predictions using the crowd, which is directly used to improve our models. This complete feedback loop, without intermediate representations or database specific engineering, opens up new ways of building high quality semantic parsers. Experiments suggest that this approach can be deployed quickly for any new target domain, as we show by learning a semantic parser for an online academic database from scratch.
1 Introduction
Existing semantic parsing approaches for building natural language interfaces to databases (NLIDBs) either use special-purpose intermediate meaning representations that lack the full expressivity of database query languages or require extensive feature engineering, making it difficult to deploy them in new domains. We present a robust approach to quickly and easily learn and deploy semantic parsers from scratch, whose performance improves over time based on user feedback and requires minimal intervention.
To learn these semantic parsers, we (1) adapt
neural sequence models to map utterances directly to SQL thereby bypassing intermediate representations and taking full advantage of SQL’s querying capabilities, (2) immediately deploy the model online to solicit questions and user feedback on results to reduce SQL annotation efforts, and (3) use crowd workers from skilled markets to provide SQL annotations that can directly be used for model improvement, in addition to being easier and cheaper to obtain than logical meaning representations. We demonstrate the effectiveness of the complete approach by successfully learning a semantic parser for an academic domain by simply deploying it online for three days.
This type of interactive learning is related to a number of recent ideas in semantic parsing, including batch learning of models that directly produce programs (e.g. regular expressions (Locascio et al., 2016)), learning from paraphrases (often gathered through crowdsourcing (Wang et al.,
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
2015)), data augmentation (e.g. based on manually engineered semantic grammars (Jia and Liang, 2016)) and learning through direct interaction with users (e.g. where a single user teaches the model new concepts (Wang et al., 2016)). However, there are unique advantages to our approach, including showing (1) that non-linguists can write SQL to encode complex, compositional computations (e.g. see Fig 1),1 (2) that external paraphrase resources and the structure of facts from the target database itself can be used for effective data augmentation, and (3) that actual database users can effectively drive the overall learning by simply providing feedback about what the model is currently getting correct.
Our experiments measure the performance of these learning advances, both in batch on existing datasets and through a simple online experiment for the full interactive setting. For the batch evaluation, we use sentences from the benchmark GeoQuery and ATIS domains, converted to contain SQL meaning representations. Our neural learning with data augmentation achieves near state-ofthe-art accuracies, despite the extra complexities of mapping directly to SQL. We also perform simulated interactive learning on this data, showing that with perfect user feedback our full approach could learn high quality parsers with only 55% of the data. Finally, we do a small scale online experiment for a new domain, academic paper metadata search, demonstrating that actual users can provide useful feedback and our full approach is an effective method for learning a high quality parser that continues to improve over time as it is used.
2 Related Work
Although diverse meaning representation languages have been used with semantic parsers – such as regular expressions (Kushman and Barzilay, 2013; Locascio et al., 2016) and systems of equations (Kushman et al., 2014; Roy et al., 2016) – parsers for querying databases have typically used either logic programs (Zelle and Mooney, 1996), lambda calculus (Zettlemoyer and Collins,
1Parsers can also be learned directly from questionanswer pairs (Liang et al., 2011). However, recent work has shown that in many domains, especially large databases, it is easier to write a query than to manually sort through the data and list the answers (Yih et al., 2016). Learning parsers directly from SQL queries has the added benefit that we can potentially hire programmers on skilled-labor crowd markets, such as UpWork, to further improve scalability, an exploration we leave to future work.
2005), or λ-DCS (Liang et al., 2011) as the meaning representation language. All three of these languages are modeled after natural language to simplify parsing. However, none of them is used to query databases outside of the semantic parsing literature; therefore, they are understood by few people and not supported by standard database implementations. In contrast, we parse directly to SQL, which is a popular database query language with wide usage and support.
A few systems have been developed to directly generate SQL queries from natural language (Popescu et al., 2003; Giordani and Moschitti, 2012; Poon, 2013). However, all of these systems make strong assumptions on the structure of queries: they use manually engineered rules that can only generate a subset of SQL, require lexical matches between question tokens and table/column names, or require questions to have a certain syntactic structure. In contrast, our approach can generate arbitrary SQL queries, only uses lexical matching for entity names, and does not depend on syntactic parsing.
We use a neural sequence-to-sequence model to directly generate SQL queries from natural language questions. This approach builds on recent work demonstrating that such models are effective for tasks such as machine translation (Bahdanau et al., 2014) and natural language generation (Kiddon et al., 2016). Recently, neural models have been successfully applied to semantic parsing with simpler meaning representation languages (Dong and Lapata, 2016; Jia and Liang, 2016) and short regular expressions (Locascio et al., 2016). Our work extends these results to the task of SQL generation. Finally, Ling et al. (2016) generate Java/Python code for trading cards given a natural language description; however, this system suffers from low overall accuracy.
A final direction of related work studies methods for reducing the annotation effort required to train a semantic parser. Semantic parsers have been trained from various kinds of annotations, including labeled queries (Zelle and Mooney, 1996; Wong and Mooney, 2007; Zettlemoyer and Collins, 2005), question/answer pairs (Liang et al., 2011; Berant et al., 2013), distant supervision (Krishnamurthy and Mitchell, 2012; Choi et al., 2015), and binary correct/incorrect feedback signals (Clarke et al., 2010). Each of these schemes presents a particular trade-off between annotation
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
effort and parser accuracy; however, recent work has suggested that labeled queries are the most effective (Yih et al., 2016). Our approach trains on fully labeled SQL queries to maximize accuracy, but uses binary feedback from users to reduce the number of queries that need to be labeled. Annotation effort can also be reduced by using crowd workers to paraphrase automatically generated questions (Wang et al., 2015); however, this approach may not generate the questions that users actually want to ask the database – an experiment in this paper demonstrated that 48% of users’ questions in a calendar domain could not be generated.
3 Feedback-based Learning
Our feedback-based learning approach can be used to quickly deploy semantic parsers to create NLIDBs for any new domain. Is is a simple interactive learning algorithm that deploys a preliminary semantic parser, then iteratively improves this parser using user feedback and selective query annotation. A key requirement of this algorithm is the ability to cheaply and efficiently annotate queries for chosen user utterances. We address this requirement by developing a model that directly outputs SQL queries (Section 4), which can also be produced by crowd workers.
Our algorithm alternates between stages of training the model and making predictions to gather user feedback, with the goal of improving performance in each successive stage. The procedure is described in Algorithm 1. Our neural model N is initially trained on synthetic data T generated by domain-independent schema templates (see Section 4), and is then ready to answer new user questions, n. The results R of executing the predicted SQL query q are presented to the user who provides a binary correct/incorrect feedback signal. If the user marks the result correct, the pair (n, q) is added to the training set. If the user marks the result incorrect, the algorithm asks a crowd worker to annotate the utterance with the correct query, q̂, and adds (n, q̂) to the training set. This procedure can be repeated indefinitely, ideally increasing parser accuracy and requesting fewer annotations in each successive stage.
4 Semantic Parsing to SQL
We use a neural sequence-to-sequence model for mapping natural language questions directly
Algorithm 1 Feedback-based learning 1: procedure LEARN(schema) 2: T ← initial data(schema) 3: while true do 4: T ← T∪ paraphrase(T ) 5: N ← train model(T ) 6: for each n ∈ new utterances do 7: q ← predict(N , n) 8: R ← execute(q) 9: f ← feedback(R) 10: if f = correct then 11: T ← T ∪ (n, q) 12: else if f = wrong then 13: q̂ ← annotate(n) 14: T ← T ∪ (n, q̂)
to SQL queries and this allows us to scale our feedback-based learning approach, by easily crowdsourcing labels when necessary. We further present two data augmentation techniques which use content from the database schema and external paraphrase resources.
4.1 Model
We use an encoder-decoder model with global attention, similar to Luong et al. (2015), where the anonymized utterance is encoded using a bidirectional LSTM network, then decoded to directly predict SQL query tokens. Fixed pre-trained word embeddings from word2vec (Mikolov et al., 2013) are concatenated to the embeddings that are learned for source tokens from the training data. The decoder predicts a conditional probability distribution over possible values for the next SQL token given the previous tokens using a combination of the previous SQL token embedding, attention over the hidden states of the encoder network, and an attention signal from the previous time step.
Formally, if qi represents an embedding for the ith SQL token qi, the decoder distribution is
p(qi|q1, . . . , qi−1) ∝ exp (W tanh(Ŵ[hi : ci])) where hi represents the hidden state output of the decoder LSTM at the ith timestep, ci represents the context vector generated using an attention weighted sum of encoder hidden states based on hi, and, W and Ŵ are linear transformations. If sj is the hidden representation generated by the encoder for the jth word in the utterance (k words
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
long), then the context vectors are defined to be
ci = k∑
j=1
αi,j · sj
The attention weights αi,j are computed using an inner product between the decoder hidden state for the current timestep hi, and the hidden representation of the jth source token sj
αi,j = exp(hiTFsj)∑k j=1 exp(hi TFsj)
where F is a linear transformation. The decoder LSTM cell f computes the next hidden state hi, and cell state, mi, based on the previous hidden and cell states, hi−1,mi−1, the embeddings of the previous SQL token qi−1 and the context vector of the previous timestep, ci−1
hi,mi = f(hi−1,mi−1,qi−1, ci−1)
We apply dropout on non-recurrent connections for regularization, as suggested by Zaremba et al. (2014). Beam search is used for decoding the SQL queries after learning.
4.2 Entity Anonymization
We handle entities in the utterances and SQL by replacing them with their types, using incremental numbering to model multiple entities of the same type (e.g., CITY NAME 1). During training, when the SQL is available, we infer the type from the associated column name; for example, Boston is a city in city.city name = ’Boston’. To recognize entities in the utterances at test time, we build a search engine on all entities from the target database. For every span of words (starting with a high span size and progressively reducing it), we query the search engine using a TF-IDF scheme to retrieve the entity that most closely matches the span, then replace the span with the entity’s type. We store these mappings and apply them to the generated SQL to fill in the entity names. TF-IDF matching allows some flexibility in matching entity names in utterances, for example, a user could say Donald Knuth instead of Donald E. Knuth.
4.3 Data Augmentation
We present two data augmentation strategies that either (1) provide the initial training data to start the interactive learning, before more labeled examples become available, or (2) use external paraphrase resources to improve generalization.
Get all <ENT1>.<NAME> having <ENT2>.<COL1>.<NAME> as <ENT2>.<COL1>.<TYPE>
SELECT <ENT1>.<DEF> FROM JOIN_FROM(<ENT1>, <ENT2>) WHERE JOIN_WHERE(<ENT1>, <ENT2>) AND <ENT2>.<COL1> = <ENT2>.<COL1>.<TYPE>
(a) Schema template
SELECT author.authorId FROM author , writes , paper , paperDataset , dataset WHERE author.authorId = writes.authorId AND writes.paperId = paper.paperId AND paper.paperId = paperDataset.paperId AND paperDataset.datasetId = dataset.datasetId AND dataset.datasetName = DATASET_TYPE
Get all author having dataset as DATASET_TYPE
(b) Generated utterance-SQL pair
Figure 2: (a) Example schema template consisting of a question and SQL query with slots to be filled with database entities, columns, and values; (b) Entity-anonymized training example generated by applying the template to an academic database.
Schema Templates To bootstrap the model to answer simple questions initially, we defined 22 language/SQL templates that are schema-agnostic, so they can be applied to any database. These templates contain slots whose values are populated given a database schema. An example template is shown in Figure 2a. The <ENT> types represent tables in the database schema, <ENT>.<COL> represents a column in the particular table and <ENT>.<COL>.<TYPE> represents the type associated with the particular column. A template is instantiated by first choosing the entities and attributes. Next, join conditions, i.e., JOIN FROM and JOIN WHERE clauses, are generated from the tables on the shortest path between the chosen tables in the database schema graph, which connects tables (graph nodes) using foreign key constraints. Figure 2b shows an instantiation of a template using the path author - writes - paper - paperdataset - dataset. SQL queries generated in this manner are guaranteed to be executable on the target database. On the language side, an English name of each entity is plugged into the template to generate an utterance for the query.
Paraphrasing The second data augmentation strategy uses the Paraphrase Database (PPDB) (Ganitkevitch et al., 2013) to automatically generate paraphrases of training utterances. Such methods have been recently used to improve performance for parsing to logical forms (Chen et al., 2016). PPDB contains over 220 million paraphrase pairs divided into 6 sets (small to XXXL)
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
based on precision of the paraphrases. We use the one-one and one-many paraphrases from the large version of PPDB. To paraphrase a training utterance, we pick a random word in the utterance that is not a stop word and replace it with a random paraphrase. We perform paraphrase expansion on all examples labeled during learning, as well as the initial seed examples from schema templates.
5 Benchmark Experiments
Our first set of experiments demonstrates that our semantic parsing model has comparable accuracy to previous work, despite the increased difficulty of directly producing SQL. We demonstrate this result by running our model on two benchmark datasets for semantic parsing, GEO880 and ATIS.
5.1 Data sets
GEO880 is a collection of 880 utterances issued to a database of US geographical facts (Geobase), originally in Prolog format. Popescu et al. (2003) created a relational database schema for Geobase together with SQL queries for a subset of 700 utterances. To compare against prior work on the full corpus, we annotated the remaining utterances and used the standard 600/280 training/test split (Zettlemoyer and Collins, 2005).
ATIS is a collection of 5,418 utterances to a flight booking system, accompanied by a relational database and SQL queries to answer the questions. We use 4,473 utterances for training, 497 for development and 448 for test, following Kwiatkowski et al. (2011). The original SQL queries were very inefficient to execute due to the use of IN clauses, so we converted them to joins (Ramakrishnan and Gehrke, 2003) while verifying that the output of the queries was unchanged.
Table 1 shows characteristics of both data sets. GEO880 has shorter queries but is more compositional: almost 40% of the SQL queries have at least one nested subquery. ATIS has the longest utterances and queries, with an average utterance length of 11 words and an average SQL query length of 67 tokens. They also operate on approximately 6 tables per query on average. We will release our processed versions of both datasets.
5.2 Experimental Methodology
We follow a standard train/dev/test methodology for our experiments. The training set is augmented using schema templates and 3 paraphrases per
training example, as described in Section 4. Utterances were anonymized by replacing them with their corresponding types and all words that occur only once were replaced by UNK symbols. The development set is used for hyperparameter tuning and early stopping. For GEO880, we use cross validation on the training set to tune hyperparameters. We used a minibatch size of 100 and used Adam (Kingma and Ba, 2014) with a learning rate of 0.001 for 70 epochs for all our experiments. We used a beam size of 5 for decoding. We report test set accuracy of our SQL query predictions by executing them on the target database and comparing the result with the true result.
5.3 Results
Tables 2 and 3 show test accuracies based on denotations for our model on GEO880 and ATIS respectively, compared with previous work.2 To our knowledge, this is the first result on directly parsing to SQL to achieve comparable performance to prior work without using any database-specific feature engineering. Popescu et al. (2003) and Giordani and Moschitti (2012) also directly produce SQL queries but on a subset of 700 examples from GEO880. The former only works on semantically tractable utterances where words can be unambiguously mapped to schema elements, while the latter uses a reranking approach that also limits the complexity of SQL queries that can be handled. GUSP (Poon, 2013) creates an intermediate representation that is then deterministically converted to SQL to obtain an accuracy of 74.8% on ATIS, which is boosted to 83.5% using manually introduced disambiguation rules. However, it requires a lot of SQL specific engineering (for example, special nodes for argmax) and is hard to extend to more complex SQL queries.
On both datasets, our SQL model has slightly
2Note that 2.8% of GEO880 and 5% ATIS gold test set SQL queries (before any processing) produced empty results.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
559
560
561
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
System Acc.
Ours (SQL) 82.5
Popescu et al. (2003) (SQL) 77.5∗ Giordani and Moschitti (2012) (SQL) 87.2∗
Dong and Lapata (2016) 84.6 † Jia and Liang (2016) 89.3 Liang et al. (2011) 91.1
Table 2: Accuracy of SQL query results on the Geo880 corpus; ∗ use Geo700; convert to logical forms instead of SQL; † measure accuracy in terms of obtaining the correct logical form, other systems use denotations
lower accuracy than the best non-SQL results. Most relevant to this work are the neural sequence based approaches of Dong and Lapata (2016) and Jia and Liang (2016). We note that Jia and Liang (2016) use a data recombination technique that boosts accuracy from 85.0 on GEO880 and 76.3 on ATIS; this technique is also compatible with our model. Our results demonstrate that these models are powerful enough to directly produce SQL queries. Thus, our methods enable us to utilize the full expressivity of the SQL language without any extensions that certain logical representations require to answer more complex queries. More importantly, it can be immediately deployed for users in new domains, with a large programming community available for annotation, and thus, fits effectively into a framework for interactive learning.
We perform ablation studies on the development sets (see Table 4) and find that paraphrasing using PPDB consistently helps boost performance. However, unlike in the interactive experiments (Section 6), data augmentation using schema templates does not improve performance in the fully supervised setting.
6 Interactive Learning Experiments
In this section, we learn a semantic parser for an academic domain from scratch by deploying an online system using our interactive learning algorithm (Section 3). After three train-deploy cycles, the system correctly answered 63.51% of user’s questions. To our knowledge, this is the first effort to learn a semantic parser using a live system, and is enabled by our models that can directly parse language to SQL without manual intervention.
6.1 User Interface
We developed a web interface for accepting natural language questions to an academic database from users, using our model to generate a SQL query, and displaying the results after execution. Several example utterances are also displayed to help users understand the domain. Together with the results of the generated SQL query, users are prompted to provide feedback which is used for interactive learning. Screenshots of our interface are included in our Supplementary Materials.
Collecting accurate user feedback on predicted queries is a key challenge in the interactive learning setting for two reasons. First, the system’s results can be incorrect due to poor entity identification or incompleteness in the database, neither of which are under the semantic parser’s control. Second, it can be difficult for users to determine if the presented results are in fact correct. This determination is especially challenging if the system responds with the correct type of result, for example, if the user requests “papers at ACL 2016” and the system responds with all ACL papers.
We address this challenge by providing users with two assists for understanding the system’s behavior, and allowing users to provide more granular feedback than simply correct/incorrect.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
The first assist is type highlighting, which highlights entities identified in the utterance, for example, “paper by Michael I. Jordan (AUTHOR) in ICRA (VENUE) in 2016 (YEAR).” This assist is especially helpful because the academic database contains noisy keyword and dataset tables that were automatically extracted from the papers. The second assist is utterance paraphrasing, which shows the user another utterance that maps to the same SQL query. For example, for the above query, the system may show “what papers does Michael I. Jordan (AUTHOR) have in ICRA (VENUE) in 2016 (YEAR).” This assist only appears if a matching query (after entity anonymization) exists in the model’s training set.
Using these assists and the predicted results, users are asked to select from five feedback options: Correct, Wrong Types, Incomplete Result, Wrong Result and Can’t Tell. The Correct and Wrong Result options represent scenarios when the user is satisfied with the result, or the result is identifiably wrong, respectively. Wrong Types indicates incorrect entity identification, which can be determined from type highlighting. Incomplete Result indicates that the query is correct but the result is not; this outcome can occur because the database is incomplete. Can’t Tell indicates that the user is unsure about the feedback to provide.
6.2 Three-Stage Online Experiment
In this experiment, using our developed user interface, we use Algorithm 1 to learn a semantic parser from scratch. The experiment had three stages; in each stage, we recruited 10 new users and asked them to issue at least 10 utterances to the system and to provide feedback on the results. We considered results marked as either Correct or Incomplete Result as correct queries for learning. The remaining incorrect utterances were sent to a crowd worker for annotation and were used to retrain the system for the next stage.
Table 5 shows the percent of utterances judged by users as either Correct or Incomplete Result in each stage. In the first stage, we do not have any labeled examples, and the model is trained using only synthetically generated data from schema templates and paraphrases (see Section 4.3). Despite the lack of real examples, the system correctly answers 25% of questions. The system’s accuracy increases and annotation effort decreases in each successive stage as additional utterances are
Feedback Error Rate (%)
Correct SQL 6.1 Incorrect SQL 6.3
Table 6: Error rates of user feedback when the SQL is correct and incorrect. The Correct and Incomplete results options are erroneous if the SQL query is correct, and vice versa for incorrect queries.
contributed and incorrect utterances are labeled. This result demonstrates that we can successfully build semantic parsers for new domains by using neural models to generate SQL with crowdsourced annotations driven by user feedback.
We analyzed the feedback signals provided by the users in the final stage of the experiment to measure the quality of feedback. We found that 22.3% of the generated queries did not execute (and hence were incorrect). 6.1% of correctly generated queries were marked wrong by users (see Table 6). This erroneous feedback results in redundant annotation of already correct examples. The main cause of this erroneous feedback was incomplete data for aggregation queries, where users chose Wrong instead of Incomplete. 6.3% of incorrect queries were erroneously deemed correct by users. It is important that this fraction be low, as these queries become incorrectly-labeled examples in the training set that may contribute to the deterioration of model accuracy over time. This quality of feedback is already sufficient for our neural models to improve with usage, and creating better interfaces to make feedback more accurate is an important task for future work.
6.3 SCHOLAR dataset
We release a new semantic parsing dataset for academic database search using the utterances gathered in the user study. We augment these labeled utterances with additional utterances labeled by crowd workers. (Note that these additional utterances were not used in the online experiment). The final dataset comprises 816 natural language utterances labeled with SQL, divided
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
0 2 4 6 8 10 12 Stages
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Fr a ct io n C o rr e ct
Simulated Interactive Learning on Geo880
Ours Without templates Without paraphrasing
1 2 3 4 5 6 7 8 9 10 Stages
0.0
0.2
0.4
0.6
0.8
1.0
Fr a ct io n C o rr e ct
Simulated Interactive Learning on ATIS
Ours Without templates Without paraphrasing
Figure 3: Accuracy as a function of batch number in simulated interactive learning experiments on Geo880 (top) and ATIS (bottom).
into a 600/216 train/test split. We also provide a database on which to execute these queries containing academic papers with their authors, citations, journals, keywords and datasets used. Table 1 shows statistics of this data set. Our parser achieves an accuracy of 67% on this train/test split in the fully supervised setting.
6.4 Simulated Interactive Experiments
We conducted additional simulated interactive learning experiments using GEO880 and ATIS to better understand the behavior of our train-deploy feedback loop, the effects of our data augmentation approaches, and the annotation effort required. We randomly divide each training set into K batches and present these batches sequentially to our interactive learning algorithm. Correctness feedback is provided by comparing the result of the predicted query to the gold query, i.e., we assume that users are able to perfectly distinguish correct results from incorrect ones.
Figure 3 shows accuracies on GEO880 and ATIS respectively of each batch when the model is trained on all previous batches. As in the live experiment, accuracy improves with successive batches. Data augmentation using templates helps
Batch Size 150 100 50
% Wrong 70.2 60.4 54.3
Table 7: Percentage of examples that required annotation (i.e., where the model initially made an incorrect prediction) on GEO880 vs. batch size.
in the initial stages of GEO880, but its advantage is reduced as more labeled data is obtained. Templates did not improve accuracy on ATIS, possibly because most ATIS queries involve two entities, i.e., a source city and a destination city, whereas our templates only generate questions with a single entity type. Nevertheless, templates are important in a live system to motivate users to interact with it in early stages. As observed before, paraphrasing improves performance at all stages.
Table 7 shows the percent of examples that require annotation using various batch sizes for GEO880. Smaller batch sizes reduce annotation effort, with a batch size of 50 requiring only 54.3% of the examples to be annotated. This result demonstrates that more frequent deployments of improved models leads to fewer mistakes.
7 Conclusion
We describe an approach to rapidly train a semantic parser as a NLIDB that iteratively improves parser accuracy over time while requiring minimal intervention. Our approach uses an attentionbased neural sequence-to-sequence model, with data augmentation from the target database and paraphrasing, to parse utterances to SQL. This model is deployed in an online system, where user feedback on its predictions is used to select utterances to send for crowd worker annotation.
We find that the semantic parsing model is comparable in performance to previous systems that either map from utterances to logical forms, or generate SQL, on two benchmark datasets, GEO880 and ATIS. We further demonstrate the effectiveness of our online system by learning a semantic parser from scratch for an academic domain. A key advantage of our approach is that it is not language-specific, and can easily be ported to other commonly- used query languages, such as SPARQL or ElasticSearch. Finally, we also release a new dataset of utterances and SQL queries for an academic domain.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

The paper presents a neural model for predicting SQL queries directly from natural language utterances, without going through an intermediate formalism. In addition, an interactive online feedback loop is proposed and tested on a small scale.  - Strengths:  1   The paper is very clearly written, properly positioned, and I enjoyed reading it.  2   The proposed model is tested and shown to perform well on 3 different domains (academic, geographic queries, and flight booking)  3   The online feedback loop is interesting and seems promising, despite of the small scale of the experiment.  4   A new semantic corpus is published as part of this work, and additionally two existing corpora are converted to SQL format, which I believe would be beneficial for future work in this area.  - Weaknesses   clarifications:  1   Section 4.2 (Entity anonymization) - I am not sure I understand the choice of the length of span for querying the search engine. Why and how is it progressively reduced? (line 333).  2   Section 5 (Benchmark experiments) - If I understand correctly, the feedback loop (algorithm 1) is *not* used for these experiments. If this is indeed the case, I'm not sure when does data augmentation occur. Is all the annotated training data augmented with paraphrases? When is the  initial data  from templates added? Is it also added to the gold training set? If so, I think it's not surprising that it doesn't help much, as the gold queries may be more diverse.  In any case, I think this should be stated more clearly. In addition, I think it's interesting to see what's the performance of the  vanilla  model, without any augmentation, I think that this is not reported in the paper.  3   Tables 2 and 3: I find the evaluation metric used here somewhat unclear.  Does the accuracy measure the correctness of the execution of the query (i.e., the retrieved answer) as the text seem to indicate? (Line 471 mentions *executing* the query). Alternatively, are the queries themselves compared? (as seems to be the case for Dong and Lapata in Table 2). If this is done differently for different systems (I.e., Dong and Lapata), how are these numbers comparable? In addition, the text mentions the SQL model has  slightly lower accuracy than the best non-SQL results  (Line 515), yet in table 2 the difference is almost 9 points in accuracy.  What is the observation based upon? Was some significance test performed? If not, I think the results are still impressive for direct to SQL parsing, but that the wording should be changed, as the difference in performance does seem significant.  4   Line 519 - Regarding the data recombination technique used in Jia and Liang (2016): Since this technique is applicable in this scenario, why not try it as well?  Currently it's an open question whether this will actually improve performance. Is this left as future work, or is there something prohibiting the use of this technique?  5   Section 6.2 (Three-stage online experiment) - several details are missing   unclear:  * What was the technical background of the recruited users?  * Who were the crowd workers, how were they recruited and trained?  * The text says  we recruited 10 new users and asked them to issue at least 10 utterances . Does this mean 10 queries *each* (e.g., 100 overall), or 10 in total (1 for each).  * What was the size of the initial (synthesized) training  set?   * Report statistics of the queries - some measure of their lexical variability   length   complexity of the generated SQL? This seems especially important for the first phase, which is doing surprisingly well. Furthermore, since SCHOLAR uses SQL and NL, it would have been nice if it were attached to this submission, to allow its review during this period.  6   Section 6.3 (SCHOLAR dataset)  * The dataset seems pretty small in modern standards (816 utterances in total), while one of the main advantages of this process is its scalability. What hindered the creation of a much larger dataset?  * Comparing performance - is it possible to run another baseline on this newly created dataset to compare against the reported 67% accuracy obtained in this paper (line 730).  7   Evaluation of interactive learning experiments (Section 6): I find the experiments to be somewhat hard to replicate as they involve manual queries of specific annotators. For example, who's to say if the annotators in the last phase just asked simpler questions? I realise that this is always problematic for online learning scenarios, but I think that an effort should be made towards an objective comparison. For starters, the statistics of the queries (as I mentioned earlier) is a readily available means to assess whether this happens. Second, maybe there can be some objective held out test set? This is problematic as the model relies on the seen queries, but scaling up the experiment (as I suggested above) might mitigate this risk. Third, is it possible to assess a different baseline using this online technique? I'm not sure whether this is applicable given that previous methods were not devised as online learning methods.  - Minor comments:  1   Line 48 -  requires  to  require   2   Footnote 1 seems too long to me. Consider moving some of its content to the body of the text.  3   Algorithm 1: I'm not sure what  new utterances  refers to (I guess it's new queries from users?). I think that an accompanying caption to the algorithm would make the reading easier.  4   Line 218 -  Is is  to  It is   5   Line 278 mentions an  anonymized  utterance. This confused me at the first reading, and if I understand correctly it refers to the anonymization described later in 4.2. I think it would be better to forward reference this.   - General Discussion:  Overall, I like the paper, and given answers to the questions I raised above, would like to see it appear in the conference.  - Author Response:  I appreciate the detailed response made by the authors, please include these details in a final version of the paper.