Other Topics You May Also Agree or Disagree: Modeling Inter-Topic Preferences using Tweets and Matrix Factorization
This paper presents an approach for modeling inter-topic preferences of Twitter users: for example, those who agree TPP also agree free trade. This kind of knowledge is useful not only for stance detection across multiple topics but also for various real-world applications including public opinion survey, election prediction, election campaign, and online debate. In order to extract users’ preferences on Twitter, we design high-quality linguistic patterns (e.g., “A is completely wrong”) in which people agree and disagree topics. By applying the linguistic patterns to a collection of tweets, we extract statements agreeing and disagreeing various topics. Inspired by the work on item recommendation, we formalize the task of modeling inter-topic preferences as matrix factorization: representing users’ preference as a user-topic matrix and mapping both users and topics into a latent feature space that abstracts the preferences. The experimental results demonstrate that the presented approach is useful for predicting missing preferences of users and that the latent vector representations of topics encode inter-topic preferences.
1 Introduction
Social media have changed the way people shape public opinions. The latest survey by Pew Research Center reported that a majority of US adults (62%) obtain news on social media, and 18% do so often (Gottfried and Shearer, 2016). As news and opinions are shared and amplified by friend networks of individuals (Jamieson and Cappella, 2008), individuals are also isolated from informa-
tion that does not fit their opinions (Pariser, 2011). Ironically, the cutting-edge technology of social media promotes ideological groups even with its potential to deliver diverse information.
A great deal of studies analyze discussions, interactions, influences, and communities on social media along the political spectrum of liberal to conservative (Adamic and Glance, 2005; Zhou et al., 2011; Cohen and Ruths, 2013; Bakshy et al., 2015; Wong et al., 2016). Even though these studies provide intuitive visualizations and interpretations along the axis of liberal-conservative, political analysts argue that the axis is flawed and insufficient for representing public opinions and ideologies (Kerlinger, 1984; Maddox and Lilie, 1984).
A potential solution for analyzing multiple axes of political spectrum on social media is stance detection (Thomas et al., 2006; Somasundaran and Wiebe, 2009; Murakami and Raymond, 2010; Anand et al., 2011; Walker et al., 2012; Mohammad et al., 2016; Johnson and Goldwasser, 2016), whose task is to determine whether the author of a text is for, neutral, or against towards a topic (e.g., free trade, immigration, abortion). However, stance detection across different topics is extremely difficult. Anand et al. (2011) reported that a sophisticated method with topic-dependent features greatly improved the performance of stance detection within a topic, but could not outperform a baseline method with simple n-gram features when evaluated across topics. More recently, all participants of SemEval 2016 Task 6A (with five topics) could not outperform the baseline supervised method using n-gram features (Mohammad et al., 2016).
In addition, stance detection encounters the difficulty with different user types. Cohen and Ruths (2013) observed that existing methods on stance detection fail on ‘ordinary’ users because they mostly obtain training and test data from polit-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
1.0
1.0
1.0 0.5
0.7
-1.0
-1.0
-0.7
-0.5
User 1
User 2
User 3
User 4
To pic
1 To pic 2 To pic 3 To pic 4
～ ～
R P T
×
Q
＝
0.9
0.9
1.0 0.5
0.7
-1.0
-1.0
-0.7
-0.5
User 1
User 2
User 3
User 4
To pic
1 To pic 2 To pic 3 To pic 4
R
-0.1
-0.1
-0.1
-0.7
0.3
-0.4
-0.9
^
Corpus (tweets) (User-topic matrix) (User vectors) (Topic vectors) (Low-rank approximation)
A good news. http://t.to/...... #TPPhantai
TPP ruins the future of our country.
............
............
............ A ruins the future of our country.
I support A A is necessary Welcome A We should introduce A ......
I disagree A A is completely wrong A ruins the future of our country ......
A good news. http://t.to/...... #TPPhantai
TPP ruins the future of our country.
............
............
............
A good news. http://t.to/...... #TPPhantai
TPP ruins the future of our country.
............
............
............
A good news. http://t.to/...... #TPPhantai
TPP ruins the future of our country.
............
............
............
Tweets posted by users who have used pro/con hastags
A is completely wrong
to A
We should introduce A
This is A
Linguistic pro/con patternsPattern candidates in which the users describe topics
Matrix factorization
Extraction of topics, users, and pattern candidates
Sort candidates and select useful patterns
Mine topic preferences
pro
con
Figure 1: A generic overview of this study.
ically vocal users (e.g., politicians); for example, they found that a stance detector trained on a dataset with politicians achieved 91% accuracy on other politicians but only 54% accuracy on ‘ordinary’ users. Establishing a bridge across different topics and users is a major challenge not only to stance detection but also to social media analytics.
An important material for the bridge is commonsense knowledge about topics. For instance, consider a topic a revision of Article 96 of the Japanese Constitution. We infer that a statement, “we should maintain armed forces,” tends to favor the topic even without any lexical overlap between the topic and statement. This inference is reasonable because: the writer of the statement favors armed forces; those who favor armed forces also favor a revision of Article 91; and those who favor a revision of Article 9 also favor a revision of Article 962. In general, this kind of commonsense knowledge can be expressed in the format: those who agree/disagree a topic A also agree/disagree another topic B. We call this kind of knowledge inter-topic preference throughout this paper.
We conjecture that the previous work on stance detection indirectly learns inter-topic preferences
1Article 9 prohibits armed forces in Japan. 2Article 96 specifies high requirements for making
amendments to Constitution of Japan (including Article 9).
within the same target through the use of n-gram features on a supervision data. In contrast, this paper directly acquires inter-topic preferences from an unlabeled corpus of tweets. The knowledge of inter-topic preferences is useful not only for stance detection but also for various real-world applications including public opinion survey, election campaign/prediction, and online debate.
Figure 1 illustrates a generic overview of this work. We extract high-quality linguistic patterns (e.g., “A is completely wrong”) in which people agree and disagree topics, making use of hashtags in a large collection of tweets (Section 2.1). The patterns are used for extracting instances of users’ preferences to various topics (Section 2.2). Inspired by the work on item recommendation, Section 3 formalizes the task of modeling inter-topic preferences as a matrix factorization: representing a sparse user-topic matrix (the extracted instances) with the product of low-rank user and topic matrices. The low-rank matrices provide latent vector representations of users and topics. This approach is also useful for completing preferences of ‘ordinary’ (less vocal) users, which fills the gap between different types of users.
The contributions of this paper are three-folds.
1. To the best of our knowledge, this is the first
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
study that models inter-topic preferences for unlimited targets on the real-world data.
2. The experimental results show that this approach can predict missing topic preferences of users accurately (80–94%).
3. The experimental results also demonstrate that the latent vector representations of topics encode inter-topic preferences, e.g., those who agree nuclear power plant also agree nuclear fuel cycle.
This study uses a Japanese Twitter corpus because of its availability from the authors, but the core idea is applicable to any language.
2 Mining topic preferences of users
This section collects statements of users agreeing or disagreeing various topics on Twitter as source data for modeling inter-topic preferences. More formally, we are interested in acquiring a collection of tuples (u, t, v), where: u ∈ U is a user; U is the set of all users on Twitter; t ∈ T is a topic; T is the set of all topics; and v ∈ {+1,−1} is +1 when the user u agree the topic t and−1 otherwise (disagreement).
Throughout this work, we use a corpus consisting of 35,328,745,115 Japanese tweets (7,340,730 users) crawled from February 6, 2013 to September 30, 2016. We removed retweets from the corpus.
2.1 Mining linguistic patterns of agreement and disagreement
We use linguistic patterns to extract tuples (u, t, v) from the corpus. More specifically, when a tweet message matches to one of linguistic patterns for agreement (e.g., “t is necessary”), we regard that the author u of the tweet agrees the topic t. Conversely, a statement of disagreement is identified by linguistic patterns for disagreement (e.g., “t is unacceptable”).
In order to design high-quality linguistic patterns, this study focuses on hashtags appearing in the corpus, which have been popular clues for locating subjective statements such as sentiments (Davidov et al., 2010), emotions (Qadir and Riloff, 2014), and ironies (Van Hee et al., 2016). Hashtags are also useful for finding strong supporters and critics and their target topics; for example, #immigrantsWelcome in-
dicates that the author favors immigrants; and #StopAbortion is against abortion.
Based on this intuition, we design regular expressions for pro hashtags “#(.+)sansei”3 and for con hashtags “#(.+)hantai”4, where (.+) matches to a target topic. These regular expressions can find users who have strong preferences to topics. In this way, we could extract 31,068 occurrences of pro/con hashtags, which were used by 18,582 users for 4,899 topics. We regard the topics found in this procedure as the set of target topics T in this study.
Every time we find a tweet containing a pro/con hashtag, we look for corresponding textual statements as follows. Suppose that a tweet mentions a hashtag (e.g., #TPPsansei) for a topic t (e.g., TPP). Assuming that the author of the tweet does not change their attitude to a topic over time, we search for other tweets posted by the same author with the topic keyword t. This process retrieves tweets like “I support TPP.” Then, we replace the topic keyword into a variableA to extract patterns, e.g., “I support A.” Here, the definition of the pattern unit is language specific. For Japanese tweets, we simply recognize a pattern starting at a variable (topic) and ending at the end of the sentence5.
Because this procedure also extracts unuseful patterns such as “to A” and “this is A”, we manually choose useful patterns in a systematic way: sort patterns in descending order of the number of users who use a pattern; and check the sorted list of patterns manually; and remove unuseful patterns. In this way, we obtain 100 pro patterns (e.g., “welcome A” and “A is necessary”) and 100 con patterns (“do not let A” and “I don’t want A”).
2.2 Extracting instances of topic preferences
By using the pro and con patterns acquired in Section 2.1, we extract instances of (u, t, v) as follows. When a sentence in a tweet (whose author is the user u) matches to one of pro patterns (e.g., “t is necessary”) and the topic t is included in the set of target topics T , we recognize this as an instance
3Unlike English hashtags, we systematically attach a noun sansei, which stands for pro (agreement) in Japanese, to a topic, for example, #TPPsansei. This paper uses the alphabetical expression sansei only for explanation; the actual pattern uses Chinese characters corresponding to sansei.
4A Japanese noun hantai stands for con (disagreement), for example, #TPPhantai. This paper uses the alphabetical expression hantai only for explanation; the actual pattern uses Chinese characters corresponding to hantai.
5In English, this treatment roughly corresponds to extracting a verb phrase with the variable A.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
of (u, t,+1). Similarly, when a sentence matches to one of con patterns (e.g., “I don’t want t”) and the topic t is included in the set of target topics T , we recognize this as an instance of (u, t,−1). In this way, we collected 25,805,909 tuples regarding 3,302,613 users and 4,899 topics. Because these collected tuples also including far less frequent users and topics, we remove users and topics appeared less than 5 times. In addition, there also meaningless frequent topics such as “of” and “it”. Therefore, we sorted topics in descending order of co-occurrence frequency with each of pro patterns and con patterns, and remove meaningless topics in the top 100 topics. This resulted in 9,961,509 tuples regarding 273,417 users and 2,323 topics.
3 Matrix factorization
Section 2 collects a number of instances of users’ preferences to various topics from the corpus. However, a Twitter user does not necessarily express preferences for all topics. In addition, it is by nature impossible to predict whether a new (non-existent in the data) user agree or disagree topics. Therefore, this section applies matrix factorization (Koren et al., 2009) in order to predict missing values, inspired by the work of item recommendation (Bell and Koren, 2007; Dror et al., 2011). In essence, matrix factorization maps both users and topics into a latent feature space that abstracts topic preferences of users.
Let R be a sparse matrix of |U |×|T |. Only when a user u expresses a preference to topic t, we compute an element of the sparse matrix ru,t,
ru,t = #(u, t,+1)−#(u, t,−1) #(u, t,+1) + #(u, t,−1)
(1)
Here, #(u, t,+1) and #(u, t,−1) are the numbers of occurrences of instances (u, t,+1) and (u, t,−1), respectively. Thus, an element ru,t approaches +1 as the user u favors the topic t, and −1 otherwise. If the user u does not make any statement about the topic t, i.e., neither (u, t,+1) nor (u, t,−1) exists in the data, we do not fill the corresponding element, leaving it as a missing value.
Matrix factorization decomposes the sparse matrix R into low dimensional matrices P ∈ Rk×|U | and Q ∈ Rk×|T |, where k is a parameter to specify the number of dimension of the latent space. We minimize the following objective function for
finding the matrices P and Q,
min P,Q ∑ (u,t)∈R ( (ru,t − puᵀqt)2
+λP ‖pu‖2 + λQ ‖qt‖2 ) . (2)
Here, (u, t) ∈ R is repeated for elements filled in the sparse matrix R, pu ∈ Rk and qv ∈ Rk are u column vectors of P , and v column vectors of Q, respectively, and λP ≥ 0 and λQ ≥ 0 present coefficients of regularization terms. We call pu and qt the user vector and topic vector, respectively.
Using the user and topic vectors, we can predict an element r̂u,t that may be missing in the original matrix R,
r̂u,t ' puᵀqt. (3)
We use libmf6 (Chin et al., 2015) for solving the optimization problem in Equation 2. We set the regularization coefficients λP = 0.1 and λQ = 0.1, and use default values for other parameters of libmf.
4 Evaluation
4.1 Determining the dimension parameter k
How good is the low-rank approximation found by matrix factorization? What is the sweet spot for the number of dimension k of the latent space? We investigate the reconstruction error of matrix factorization with different values of k in order to answer these questions. We use Root Mean Squared
6https://github.com/cjlin1/libmf
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Matrix factorization
Majority baseline
Figure 3: Prediction accuracy when changing the threshold for the number of known topic preferences of each user.
Error (RMSE) for measuring the error,
RMSE =
√∑ (u,t)∈R (pu ᵀqt − ru,t)2
N . (4)
Here, N is the number of elements in the sparse matrix R (the number of known values).
Figure 2 shows RMSE values over iterations of libmf with the dimension parameter k ∈ {1, 2, 5, 10, 30, 50, 100, 300, 500}. We can observe that a reconstruction error decreases as the iterative method of libmf progresses. The larger the number of dimension k is, the less the reconstruction error becomes; the lowest reconstruction error was 0.3256 with k = 500. It is also interesting to observe the error with k = 1, which corresponds to mapping users and topics onto one dimension similarly to the political spectrum of liberal and conservative. Judging from the relatively high RMSE values with k = 1, it may be difficult to represent everything in the data on a onedimensional axis. Based on this result, we draw a conclusion that matrix factorization with k = 100 is sufficient to reconstruct the original matrix R, and use this parameter value in the rest of the experiments.
4.2 Predicting missing topic preferences
How accurately can the user and topic vectors predict missing topic preferences? In order to investigate this question, we evaluate the accuracy for predicting hidden preferences in the matrix R as follows. First, we randomly select 5% of existing elements in R, and let Y the collection of the selected elements (test set). We then perform matrix factorization on the sparse matrix without the selected elements of Y and only with the remaining
95% elements of R (training set). The accuracy of the prediction is defined by,
1 |Y | ∑ u,t∈Y 1 (sign(r̂u,t) = sign(ru,t)) (5)
Here, ru,t denotes the actual (self-declared) preference values, r̂u,t is the preference value predicted by Equation 3, sign(.) presents the sign of the argument, and 1(.) yields 1 only when the condition described in the argument holds and 0 otherwise. In other words, Equation 5 computes the proportion of correct predictions, assuming zero to be the decision boundary between pro and con.
Figure 3 plots prediction accuracy values calculated from different sets of users. Here, the x-axis presents a threshold θ that filters out users whose declarations of topic preferences are no greater than θ topics. In other words, Figure 3 shows prediction accuracy when we know users’ preferences for at least θ topics. For comparison, we also include the majority baseline that predicts pro and con based on the majority of preferences about each topic in the training set.
The presented method could predict missing preferences with 82.1% accuracy for users stating preferences for at least 5 topics. The accuracy increases as the method receives more information about the users, reaching 94.0% accuracy when θ = 100. This result again indicates that the presented method reasonably utilize known preferences to complete missing preferences.
In contrast, the majority baseline decreases its performance as it receives more information about the users. Because this result was counterintuitive, we examined the cause of this phenomenon. Consequently, this result turned out reasonable because preferences of vocal users deviate
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
from those of the average users. Figure 4 illustrates this finding, showing the mean of variances of preference values ru,t of all topics. The x-axis presents a threshold θ that filters out users whose statements of topic preferences are no greater than θ topics. We can observe that the mean variance increases as we focus on vocal users. These results demonstrate the usefulness of user and topic vectors for predicting missing preferences.
Table 1 shows examples where missing preferences of two users are predicted from known statements of agreements and disagreements7. A predicted topic accompanies with the value of r̂u,t in parentheses. For example, the proposed method predicts that the user A, who is positive to regime change but negative to Okinawa US military base, may also be positive to vote of non-confidence to Cabinet but negative to construction of a new base.
4.3 Inter-topic preferences
Does the topic vectors obtained by matrix factorization capture inter-topic preferences such as “People who agree with A also agree with B”?
Because no dataset exists for this evaluation, we created a dataset of pairwise inter-topic preferences by using a crowdsourcing service8. Sampling topic pairs randomly, we collected 150 topic pairs whose cosine similarities of topic vectors are below −0.6, 150 pairs whose cosine similarities are between −0.6 and 0.6, and 150 pairs whose cosine similarities are above 0.6. In this way, we obtained 450 topic pairs for evaluation.
Given a pair of topics A and B, a crowd worker is asked to choose a label from: (a) those who agree/disagree a topic A may also agree/disagree topic B; (b) those who agree/disagree a topic A may conversely disagree/agree topic B; (c) otherwise (no association between A and B). Creating twenty pairs of topics as a gold data, we removed labeling results from workers whose accuracy is lower than 90%.
Consequently, we obtained 6–10 human judgements for every topic pair. Regarding (a) as +1 point, (b) as −1 point, and (c) as 0 point, we com-
7We anonymized user names in these examples. In addition, we removed topics that are too discriminatory or aggressive to other countries and races. Even though the experimental results of this paper do not necessarily reflect our idea, we do not think it is a good idea to distribute politically incorrect ideas through this paper.
8We used Yahoo! Crowdsourcing, a Japanese online service for crowdsourcing. http://crowdsourcing.yahoo.co.jp/
pute the mean of the points (as the average human judgements) for every topic pair. Spearman’s rank correlation coefficient (ρ) between cosine similarity values of topic vectors and human judgements was 0.2210. We could observe a moderate correlation even though inter-topic preferences collected in this manner are highly subjective.
In addition to the quantitative evaluation, we also check similar topics for three controversial topics, Liberal Democratic Party (LDP), constitutional amendment and right of foreigners to vote (Table 2). The similar topics to Liberal Democratic Party (LDP) were synonymous ones (e.g., Abe’s LDP, Abe administration) as well as other topics promoted by the LDP (e.g., resuming nuclear power plant operations, bus rapid transit (BRT), hate speech countermeasure law). Considering that people who support the LDP may also tend to favor its policies, we found these results reasonable. In the other example, constitutional amendment has a similar feature vector to amendment of Article 9, enforcement of specific secrete protection law and security related law. From these results, we conclude that topic vectors were able to capture inter-topic preferences.
5 Related work
This section summarizes the related work that spreads across various research fields.
Social science and political science A number of of studies analyze social phenomena regarding political activities, political thoughts, and public opinions on social media. These studies model the political spectrum of liberal to conservative (Adamic and Glance, 2005; Zhou et al., 2011; Cohen and Ruths, 2013; Bakshy et al., 2015; Wong et al., 2016), political parties (Tumasjan et al., 2010; Boutet et al., 2013; Makazhanov and Rafiei, 2013), and elections (O’Connor et al., 2010; Conover et al., 2011).
Employing a single axis (e.g., liberal to conservative) or a few axes (e.g., political parties and candidates of elections), these studies provide intuitive visualizations and interpretations along the axes. In contrast, this study is the first attempt for recognizing and organizing various axes of topics on social media with no prior assumption about axes. Therefore, we think this study provides a new tool for computational social science and political science to analyze and interpret phenomena on social media.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
User Type Topic A Agreement (declared) regime change, capital relocation
Disagreement (declared) Okinawa US military base, nuclear weapons, TPP, Abe Cabinet, Abe government, nuclear cycle, right to collective defense, nuclear power plant, Abenomics Agreement (predicted) same-sex partnership ordinance (0.9697), vote of non-confidence to Cabinet (0.9248), national people’s government (0.9157), abolition of tax (0.8978) Disagreement (predicted) steamrollering war bill (-1.0522), worsening dispatch law (-1.0301), Sendai nuclear power plant (-1.0269), war bill (-1.0190), constructing new base (-1.0186), Abe administration (-1.0173), landfill Henoko (-1.0158), unreasonable arrest (-1.0113)
B Agreement (declared) visit shrine, marriage Disagreement(declared) tax increase, conscription, amend Article 9 Agreement (predicted) national people’s government (0.8467), abolition of tax (0.8300), same-sex partner-
ship ordinance (0.7700), security bills (0.6736) Disagreement (predicted) corporate tax cuts (-1.0439), Liberal Democratic Party’s draft constitution (-1.0396),
radioactivity (-1.0276), rubble (-1.0159), nuclear cycle (-1.0143)
Table 1: Examples of agreement/disagreement topics predicted for each user (predicted score r̂u,v is shown in parenthesis).
Topic Topics with a high degree of cosine similarity Liberal Democratic Party (LDP) Abe’s LDP (0.3937), resuming nuclear power plant operations (0.3765), bus rapid
transit (BRT) (0.3410), hate speech countermeasure law (0.3373), Henoko relocation (0.3353), C-130 (0.3338), Abe administration (0.3248), LDP & Komeito (0.2898), Prime Minister Abe (0.2835)
constitutional amendment amendment of Article 9 (0.4520), enforcement of specific secrete protection law (0.4399), security related law (0.4242), specific confidentiality protection law (0.4022), security bill amendment (0.3977), defense forces (0.3962), my number law (0.3874), collective self-defense rights (0.3687), militarist revival (0.3567) right of foreigners to vote human rights law (0.5405), anti-discrimination law (0.5376), hate speech countermeasure law (0.5080), foreigner’s life protection (0.4553), immigration refugee (0.4520), co-organized Olympics (0.4379)
Table 2: Similar topics for three controversial topics.
Here, we mention some work that acquires lexical knowledge about politics. Sim et al. (2013) measured ideological positions of candidates of US Presidential elections from their speeches. The study first constructs “cue lexicons” from political writings labeled with ideologies by domain experts, using sparse additive generative models (Eisenstein et al., 2011). The constructed cue lexicons are associated with ideologies such as left, center, and right. Representing each speech of a candidate with cue lexicons, they infer the proportions of ideologies of the candidate. The study requires a predefined set of labels and text data associated with the labels.
Bamman and Smith (2015) presented an unsupervised method for assessing the political stance of a proposition like “global warminig is a hoax,” along the political spectrum of liberal to conservative. In their work, a proposition is represented by a tuple in the form of 〈subject, predicate〉, e.g., 〈global warminig, hoax〉. They presented a generative model for users, subjects, and predicates to find a one-dimensional latent space, which corre-
sponds to the political spectrum. Similarly to ours, their work (Bamman and Smith, 2015) does not requires labeled data for mapping users and topics (subjects) into a latent feature space. Their paper reported that the generative model outperformed Principal Component Analysis (PCA), which is a method for matrix factorization. The empirical result probably reflected the underlying assumptions that PCA treats missing elements as zero (not as simply missing data). In contrast, this work properly distinguishes missing values from zero, excluding missing elements of the original matrix from the objective function of Equation 2. Furthermore, this work demonstrated the usefulness of the latent space, i.e., topic and user vectors, for predicting missing topic preferences of users and inter-topic preferences.
Fine-grained opinion analysis The method presented in Section 2 is an instance of finegrained opinion analysis (Wiebe et al., 2005; Choi et al., 2006; Johansson and Moschitti, 2010; Yang and Cardie, 2013; Deng and Wiebe, 2015), which extracts a tuple of a subjective opinion, a holder of
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
the opinion, and a target of the opinion from text. Although these previous studies have a potential to improve the quality of the user-topic matrix R, unfortunately, no corpus nor resource is available for Japanese language. We do not have a large collection of English tweets at this moment, but combining fine-grained opinion analysis with matrix factorization is an immediate future work.
Causality relation Some of inter-topic preferences in this work can be explained by causality relation, for example, “TPP promotes free trade.” A number of previous studies acquire instances of causal relation (Girju, 2003; Do et al., 2011) and promote/supress relation (Hashimoto et al., 2012; Fluck et al., 2015) from text. The causality knowledge is useful for predicting (hypotheses of) future events (Radinsky et al., 2012; Radinsky and Davidovich, 2012; Hashimoto et al., 2015).
However, inter-topic preferences also include pairs of topics where causality relation hardly holds. For example, it is unreasonable to infer that nuclear plant and railroading of bills have causal relation, but those who dislike nuclear plant also oppose railroading of bills because (they think) the governing political parties rush the bill for resuming a nuclear plant. This study models this kind of inter-topic preferences based on the preferences of the public. Having said that, it is a promising future direction of this work to incorporate the approach for acquiring causality knowledge.
6 Conclusion
This paper presents a novel approach for modeling inter-topic preferences of users on Twitter. Designing high-quality linguistic patterns for locating support and opposition statements, we extracted users’ preferences to various topics from a large collection of tweets. We formalized the task of modeling inter-topic preferences as a matrix factorization that maps both users and topics into a latent feature space that abstracts users’ preferences. The experimental results demonstrate that this approach can predict missing topic preferences of users accurately (80–94%) and that the latent vector representations of topics encode intertopic preferences.
An immediate future work is to embed the topic and user vectors to a cross-topic stance detector. It is possible to generalize this work for modeling heterogeneous signals such as interests and behav-
ior of people, for example, “those who are interested in A also support B,” and “those who favor A also vote forB”. Thus, we believe that this work will bring a new application of NLP to other disciplines.
Acknowledgements
(Removed for the reviewing process)

- Strengths:  The deviation between  vocal  users and  average users  is an interesting discovery that could be applied as a way to identify different types of users.  - Weaknesses:  I see it as an initial work on a new topic that should be expanded in the future. A possible comparison between matrix factorization and similar topics  in distributional semantics (e.g. latent semantic analysis) would be useful.   - General Discussion:  In this paper, the authors describe an approach for modeling the stance sentiment of Twitter users about topics. In particular, they address the task of inter-topic preferences modeling. This task consists of measuring the degree to which the stances about different topics are mutually related.This work is claimed to advance state of the art in this task, since previous works were case studies, while the proposed one is about unlimited topics on real-world data.The adopted approach consists of the following steps: A set of linguistic patterns was manually created and, through them, a large number of tweets expressing stance towards various topics was collected. Next, the texts were expressed as triples containing user, topic, and evaluation. The relationships represented by the tuples were arranged as a sparse matrix. After matrix factorization, a low-rank approximation was performed. The optimal rank was identified as 100. The definition of cosine similarity is used to measure the similarity between topics and, thus, detect latent preferences not represented in the original sparse matrix. Finally, cosine similarity is also used to detect inter-topic preferences.A preliminary empirical evaluation shows that the model predicts missing topics preferences. Moreover, predicted inter-topic preferences moderately correlate with the corresponding values from a crowdsourced gold-standard collection of preferences.  According to the overview discussed in the related work section, there are no previous systems to be compared in the latter task (i.e. prediction of inter-topic preferences) and, for this reason, it is promising.  I listed some specific comments below.  - Rows 23 and 744,  high-quality : What makes them high-quality? If not properly defined, I would remove all the occurrences of  high-quality  in the paper.  - Row 181 and caption of Figure 1: I would remove the term  generic.   - Row 217,  This section collect : to  We collected  or  This section explains how we collected - Row 246:  ironies  to  irony   - Row 269,  I support TPP : Since the procedure can detect various patterns such as  to A  or  this is A,  maybe the author should explain that all possible patterns containing the topic are collected, and next manually filtered?  - Rows 275 and 280,  unuseful : to useless  - Row 306,  including : to are including  - Row 309:   of  or  it  are not topics but, I guess, terms retrieved by mistakes as topics.   - Rows 317-319: I would remove the first sentence and start with  Twitter user...   - Rows 419-439:  I like the procedure used to find the optimal k. In previous works, this number is often assumed, while it is useful to find it empirically.   - Row 446,  let : Is it  call ?


Joint Extraction of Entities and Relations Based on a Novel Tagging Scheme
Joint extraction of entities and relations is an important task in information extraction. To tackle this problem, we firstly propose a novel tagging scheme that can convert the joint extraction task to a tagging problem. Then, based on our tagging scheme, we study different end-toend models to extract entities and their relations directly, without identifying entities and relations separately. We conduct experiments on a public dataset produced by distant supervision method and the experimental results show that the tagging based methods are better than most of the existing pipelined and joint learning methods. What’s more, the end-to-end model proposed in this paper, achieves the best results on the public dataset.
1 Introduction
Joint extraction of entities and relations is to detect entity mentions and recognize their semantic relations simultaneously from unstructured text, as Figure 1 shows. Different from open information extraction (Open IE) (Banko et al., 2007) whose relation words are extracted from the given sentence, in this task, relation words are extracted from a predefined relation set which may not appear in the given sentence. It is an important issue in knowledge extraction and automatic construction of knowledge base.
Traditional methods handle this task in a pipelined manner, i.e., extracting the entities (Nadeau and Sekine, 2007) first and then recognizing their relations (Rink, 2010). This separated framework makes the task easy to deal with, and each component can be more flexible. But it neglects the relevance between these two sub-tasks
and each subtask is an independent model. The results of entity recognition may affect the performance of relation classification and lead to erroneous delivery (Li and Ji, 2014).
Different from the pipelined methods, joint learning framework is to extract entities together with relations using a single model. It can effectively integrate the information of entities and relations, and it has been shown to achieve better results in this task. However, most existing joint methods are feature-based structured systems (Li and Ji, 2014; Miwa and Sasaki, 2014; Yu and Lam, 2010; Ren et al., 2017). They need complicated feature engineering and heavily rely on the other NLP toolkits, which might also lead to error propagation. In order to reduce the manual work in feature extraction, recently, (Miwa and Bansal, 2016) presents a neural networkbased method for the end-to-end entities and relations extraction. Although the joint models can represent both entities and relations with shared parameters in a single model, they also extract the entities and relations separately and produce redundant information. For instance, the sentence in Figure 1 contains three entities: “United States”, “Obama” and “Apple Inc”. But only “United States” and “Obama” hold a fix relation “CountryPresident”. Entity “Apple Inc” has no obvious relationship with the other entities in this sentence. Hence, the extracted result from this sentence is {United Statese1, Country-Presidentr,
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Obamae2}, which called triplet here. In this paper, we focus on the extraction of triplets that are composed of two entities and one relation between these two entities. Therefore, we can model the triplets directly, rather than extracting the entities and relations separately. Based on the motivations, we propose a tagging scheme accompanied with the end-to-end model to settle this problem. We design a kind of novel tags which contain the information of entities and the relationships they hold. Based on this tagging scheme, the joint extraction of entities and relations can be transformed into a tagging problem. In this way, we can also easily use neural networks to model the task without complicated feature engineering.
Recently, end-to-end models based on LSTM (Hochreiter and Schmidhuber, 1997) have been successfully applied to various tagging tasks: Named Entity Recognition (Lample et al., 2016), CCG Supertagging (Vaswani et al., 2016), Chunking (Zhai et al., 2017) et al. LSTM is capable of learning long-term dependencies, which is beneficial to sequence modeling tasks. Therefore, based on our tagging scheme, we investigate different kinds of LSTM-based end-to-end models to jointly extract the entities and relations. We also modify the decoding method by adding a bias loss to make it more suitable for our special tags.
The method we proposed is a supervised learning algorithm. In reality, however, the process of manually labeling a training set with a large number of entity and relation is too expensive and error-prone. Therefore, we conduct experiments on a public dataset1 which is produced by distant supervision method (Ren et al., 2017) to validate our approach. The experimental results show that our tagging scheme is effective in this task. In addition, our end-to-end model can achieve the best results on the public dataset.
The major contributions of this paper are: (1) A novel tagging scheme is proposed to jointly extract entities and relations, which can easily transform the extraction problem into a tagging task. It is the first work to solve the problem by a tagging manner. (2) Based on our tagging scheme, we study different kinds of end-to-end models to settle the problem. The tagging-based methods are better than most of the existing pipelined and joint learning methods. (3) Furthermore, we also develop an end-to-end model with a bias objective
1https://github.com/shanzhenren/CoType
function to suit for the novel tags. It can enhance the association between related entities.
2 Related Works
Entities and relations extraction is an important step to construct a knowledge base, which can be benefit for many NLP tasks (Zou et al., 2014). Two main frameworks have been widely used to solve the problem of extracting entity and their relationships. One is the pipelined method and the other is the joint learning method. The pipelined method treats this task as two separated tasks, i.e., named entity recognition (NER) (Nadeau and Sekine, 2007) and relation classification (RC) (Rink, 2010). Classical NER models are linear statistical models, such as Hidden Markov Models (HMM) and Conditional Random Fields (CRF) (Passos et al., 2014; Luo et al., 2015). Recently, several neural network architectures (Chiu and Nichols, 2015; Huang et al., 2015; Lample et al., 2016) have been successfully applied to NER, which is regarded as a sequential token tagging task. Existing methods for relation classification can also be divided into handcrafted feature based methods (Rink, 2010; Kambhatla, 2004) and neural network based methods (Xu, 2015a; Zheng et al., 2016; Zeng, 2014; Xu, 2015b; dos Santos, 2015). While joint models extract entities and relations using a single model. Most of the joint methods are feature-based structured systems (Ren et al., 2017; Yang and Cardie, 2013; Singh et al., 2013; Miwa and Sasaki, 2014; Li and Ji, 2014). Recently, (Miwa and Bansal, 2016) uses a LSTM-based model to extract entities and relations, which can reduce the manual work.
Different from the above methods, the method proposed in this paper is based on a special tagging manner, so that we can easily use end-toend model to extract results without NER and RC. end-to-end method is to map the input sentence into meaningful vectors and then back to produce a sequence. It is widely used in machine translation (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014) and sequence tagging tasks (Lample et al., 2016; Vaswani et al., 2016). Most methods apply bidirectional LSTM to encode the input sentences, but the decoding methods are always different. For examples, (Lample et al., 2016) use a CRF layers to decode the tag sequence, while (Vaswani et al., 2016; Katiyar and Cardie, 2016) apply LSTM layer to produce the tag sequence.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Input Sentence: The United States President Obama will visit the Apple Inc founded by Steven Paul Jobs
{Apple Inc, Company-Founder, Steven Paul Jobs}Final Results:
Tags: O B-CP-1 E-CP-1 O S-CP-2 O O O B-CF-1 E-CF-1 O O B-CF-2 I-CF-2 E-CF-2
{United States, Country-President, Obama}
Figure 2: Gold standard annotation for an example sentence based on our tagging scheme, where “CP” is short for “Country-President” and “CF” is short for “Company-Founder”.
3 Method
We propose a novel tagging scheme and an end-toend model with biased objective function to jointly extract entities and their relations. In this section, we firstly introduce how to change the extraction problem to a tagging problem based on our tagging method. Then we detail the model we used to extract results.
3.1 The Tagging Scheme
Figure 2 is an example of how the results are tagged. Each word is assigned a label that contributes to extract the results. Tag “O” represents the “Other” tag, which means that the corresponding word is independent of the extracted results. In addition to “O”, the other tags consist of three parts: the word position in the entity, the relation type, and the relation role. We use the “BIES” (Begin, Inside, End,Single) signs to represent the position information of a word in the entity. The relation type information is obtained from a predefined set of relations and the relation role information is represented by the numbers “1” and “2”. An extracted result is represented by a triplet: (Entity1, RelationType,Entity2). “1” means that the word belongs to the first entity in the triplet, while “2” belongs to second entity that behind the relation type. Thus, the total number of tags is Nt = 2 ∗ 4 ∗ |R|+ 1, where |R| is the size of the predefined relation set.
Figure 2 is an example illustrating our tagging method. The input sentence contains two triplets: {United States, CountryPresident, Obama} and {Apple Inc, CompanyFounder, Steven Paul Jobs}, where “CountryPresident” and “Company-Founder” are the predefined relation types. The words “United”,“States”,“Obama”,“Apple”,“Inc” ,“Steven”, “Paul” and “Jobs” are all related to the final extracted results. Thus they are tagged based
on our special tags. For example, the word of “United” is the first word of entity “United States” and is related to the relation “Country-President”, so its tag is “B-CP-1”. The other entity “Obama”, which is corresponding to “United States”, is labeled as “S-CP-2”. Besides, the other words irrelevant to the final result are labeled as “O”.
3.2 From Tag Sequence To Extracted Results
From the tag sequence in Figure 2, we know that “Obama” and “United States” share the same relation type “Country-President”, “Apple Inc” and “Steven Paul Jobs” share the same relation type “Company-Founder”. We combine entities with the same relation type into a triplet to get the final result. Accordingly, “Obama” and “United States” can be combined into a triplet whose relation type is “Country-President”. Because, the relation role of “Obama” is “2” and “United States” is “1”, the final result is {United States, CountryPresident, Obama}. The same applies to {Apple Inc, Company-Founder, Steven Paul Jobs}.
Besides, if a sentence contains two or more triplets with the same relation type, we combine every two entities into a triplet based on the nearest principle. For example, if the relation type “Country-President” in Figure 2 is “CompanyFounder”, then there will be four entities in the given sentence with the same relation type. “United States” is closest to entity “Obama” and the “Apple Inc” is closest to “Jobs”, so the results will be {United States, Company-Founder, Obama} and {Apple Inc, Company-Founder, Steven Paul Jobs}. In this paper, we only consider the situation where an entity belongs to a triplet, and we leave identification of overlapping relations for future work.
3.3 The end-to-end Model
In recent years, end-to-end model based on neural network is been widely used in sequence tagging
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
task. In this paper, we investigate an end-to-end Model to produce the tags sequence. It contains a bi-directional Long Short Term Memory (BiLSTM) layer to encode the input sentence and a LSTM decoding layer with bias loss. The bias loss can enhance the relevance of entity tags. The Bi-LSTM Encoding Layer. In sequence tagging problems, the Bi-LSTM encoding layer has been shown the effectiveness to capture the semantic information of each word. It contains word embedding layer, forward lstm layer, backward lstm layer and the concatenate layer. The word embedding layer converts the word with 1- hot representation to an embedding vector. Hence, a sequence of words can be represented as W = {w1, ...wt, wt+1...wn}, where wt ∈ Rd is the ddimensional word vector corresponding to the t-th word in the sentence and n is the length of the given sentence. After word embedding layer, there are two parallel LSTM layers: forward lstm layer and backward lstm layer. For each word wt, the forward layer will encode wt by considering the contextual information from wordw1 towt, which is marked as −→ ht . In the similar way, the backward layer will encodewt based on the contextual information from wn to wt, which is marked as ←− ht .
The LSTM architecture consists of a set of recurrently connected subnets, known as memory blocks. Each time-step is a LSTM memory block, which is used to compute current hidden vector ht based on the previous hidden vector ht−1, the previous cell vector ct−1 and the current input word embedding wt. It can be shortly denoted as: −→ ht = lstm( −−→ ht−1,
−−→ct−1, wt) and←− ht = lstm( ←−− ht+1,
←−−ct+1, wt). The detail operations are defined as follows:
it = δ(Wxixt +Whiht−1 +Wcict−1 + bi), (1)
ft = δ(Wxfxt +Whfht−1 +Wcfct−1 + bf ), (2)
zt = tanh(Wxcxt +Whcht−1 + bc), (3)
ct = ftct−1 + itzt, (4)
ot = δ(Wxoxt +Whoht−1 +Wcoct + bo), (5)
ht = ottanh(ct), (6)
where i, f and o are the input gate, forget gate and output gate respectively, b is the bias term, c is the cell memory, · denotes element-wise multiplication and W(.) are the parameters. Finally, we
concatenate ←− ht and −→ ht to represent word t’s encoding information, denoted as ht = [ ←− ht , −→ ht ]. The LSTM Decoding Layer. We also adopt a LSTM structure to produce the tag sequence. When detecting the tag of word wt, the inputs of decoding layer are: ht obtained from Bi-LSTM encoding layer, former predicted tag vector Tt−1, and the former hidden state of decoding LSTM st−1. Each unit of the decoding LSTM is the same as the encoding lstm memory block except for the input gate, which can be rewritten as:
it = δ(Wxiht +Whist−1 +WtiTt−1 + bi), (7)
where the tag embedding T is transformed from the hidden state s as follows:
Tt = Wtsst + bts. (8)
The final softmax layer computes normalized entity tag probabilities based on the tag predicted vector Tt:
yt = WyTt + by, (9)
pit = exp(yit) Nt∑ j=1 exp(yjt ) , (10)
where Wy is the softmax matrix, Nt is the total number of tags. Because the T is similar to tag embedding and LSTM is capable of learning longterm dependencies, the decoding manner can model tag interactions. The Bias Objective Function. We train our model to maximize the log-likelihood of the data and the optimization method we used is RMSprop proposed by Hinton in (Tieleman and Hinton, 2012). The objective function can be defined as:
L =max |D|∑ j=1 Lj∑ t=1 (log(p (j) t = y (j) t |xj ,Θ) · I(O)
+α · log(p(j)t = y (j) t |xj ,Θ) · (1− I(O))),
where |D| is the size of training set, Lj is the length of sentence xj , y (j) t is the label of word t in sentence xj and p (j) t is the normalized probabilities of tags which defined in Formula 10. Besides, I(O) is a switching function to distinguish the loss of tag ’O’ and relational tags that can indicate the results. It is defined as follows:
I(O) =
{ 1, if tag = ′O′
0, if tag 6= ′O′.
α is the bias weight. The larger α is, the greater influence of relational tags on the model.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
4 Experiments
4.1 Experimental setting
Dataset To evaluate the performance of our methods, we use the public dataset NYT 2 which is produced by distant supervision method (Ren et al., 2017). A large amount of training data can be obtained by means of distant supervision methods without manually labeling. While the test set is manually labeled to ensure its quality. In total, the training data contains 1.18M sentences samples from New York Times News with 353k triplets. The test set contains 395 manually labeling sentences with 3, 880 triplets. Besides, the size of relation set is 24. Evaluation We adopt standard Precision (Prec), Recall (Rec) and F1 score to evaluate the results. A triplet is regarded as correct when its relation type and the head offsets of two corresponding entities are both correct. Besides, the ground-truth relation mentions are given and “None” label is excluded as (Ren et al., 2017; Li and Ji, 2014; Miwa and Bansal, 2016) did. We create a validation set by randomly sampling 10% data from test set and use the remaining data as evaluation based on (Ren et al., 2017)’s suggestion. We run 10 times for each experiment then report the average results and their standard deviation as Table 1 shows. Hyperparameters Our model consists of a BiLSTM encoding layer and a LSTM decoding layer with bias objective function. The word embeddings used in the encoding part are initialed by running word2vec3 (Mikolov et al., 2013) on NYT training corpus. The dimension of the word embeddings is d = 300. We regularize our network using dropout on embedding layer and the dropout ratio is 0.5. The number of lstm units in encoding layer is 300 and the number in decoding layer is 600. The bias parameter α corresponding to the results in Table 1 is 10. Baselines We compare our method with several classical triplet extraction methods, which can be divided into the following categories: the pipelined methods, the jointly extracting method-
2The dataset can be downloaded at: https://github.com/shanzhenren/CoType. There are three data sets in the public resource and we only use the NYT dataset. Because more than 50% of the data in BioInfer has overlapping relations which is beyond the scope of this paper. As for dataset Wiki-KBP, the number of relation type in the test set is more than that of the train set, which is also not suitable for a supervised training method.
3https://code.google.com/archive/p/word2vec/
s and the end-to-end methods based our tagging scheme.
For the pipelined methods, we follow (Ren et al., 2017)’s settings: The NER results are obtained by CoType (Ren et al., 2017) then several classical relation classification methods are applied to detect the relations. These methods are: (1) DS-logistic (Mintz et al., 2009) is a distant supervised and feature based method, which combines the advantages of supervised IE and unsupervised IE features; (2) LINE (Tang et al., 2015) is a network embedding method, which is suitable for arbitrary types of information networks; (3) FCM (Gormley et al., 2015) is a compositional model that combines lexicalized linguistic context and word embeddings for relation extraction.
The jointly extracting methods used in this paper are listed as follows: (4) DS-Joint (Li and Ji, 2014) is a supervised method, which jointly extracts entities and relations using structured perceptron on human-annotated dataset; (5) MultiR (Hoffmann et al., 2011) is a typical distant supervised method based on multi-instance learning algorithms to combat the noisy training data; (6) CoType (Ren et al., 2017) is a domain independent framework by jointly embedding entity mentions, relation mentions, text features and type labels into meaningful representations.
In addition, we also compare our method with two classical end-to-end tagging models: LSTMCRF (Lample et al., 2016) and LSTM-LSTM (Vaswani et al., 2016). LSTM-CRF is proposed for entity recognition by using a bidirectional LSTM to encode input sentence and a conditional random fields to predict the entity tag sequence. Different from LSTM-CRF, LSTM-LSTM uses a LSTM layer to decode the tag sequence instead of CRF. They are used for the first time to jointly extract entities and relations based on our tagging scheme.
4.2 Experimental Results
We report the results of different methods as shown in Table 1. It can be seen that our method, LSTM-LSTM-Bias, outperforms all other methods in F1 score and achieves a 3% improvement in F1 over the best method CoType (Ren et al., 2017). It shows the effectiveness of our proposed method. Furthermore, from Table 1, we also can see that the jointly extracting methods are better than pipelined methods, and the tagging methods
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Methods Prec. Rec. F1 FCM 0.553 0.154 0.240 DS+logistic 0.258 0.393 0.311 LINE 0.335 0.329 0.332
MultiR 0.338 0.327 0.333 DS-Joint 0.574 0.256 0.354 CoType 0.423 0.511 0.463
LSTM-CRF 0.693± 0.008 0.310± 0.007 0.428± 0.008 LSTM-LSTM 0.682± 0.007 0.320± 0.006 0.436± 0.006 LSTM-LSTM-Bias 0.615± 0.008 0.414± 0.005 0.495± 0.006
Table 1: The predicted results of different methods on extracting both entities and their relations. The first part (from row 1 to row 3) is the pipelined methods and the second part (row 4 to 6) is the jointly extracting methods. Our tagging methods are shown in part three (row 7 to 9). In this part, we not only report the results of precision, recall and F1, we also compute their standard deviation.
Elements E1 E2 (E1,E2) PRF Prec. Rec. F1 Prec. Rec. F1 Prec. Rec. F1
LSTM-CRF 0.596 0.325 0.420 0.605 0.325 0.423 0.724 0.341 0.465 LSTM-LSTM 0.593 0.342 0.434 0.619 0.334 0.434 0.705 0.340 0.458 LSTM-LSTM-Bias 0.590 0.479 0.529 0.597 0.451 0.514 0.645 0.437 0.520
Table 2: The predicted results of triplet’s elements based on our tagging scheme.
are better than most of the jointly extracting methods. It also validates the validity of our tagging scheme for the task of jointly extracting entities and relations.
When compared with the traditional methods, the precisions of the end-to-end models are significantly improved. But only LSTM-LSTM-Bias can be better to balance the precision and recall. The reason may be that these end-to-end models all use a Bi-LSTM encoding input sentence and different neural networks to decode the results. The methods based on neural networks can well fit the data. Therefore, they can learn the common features of the training set well and may lead to the lower expansibility. We also find that the LSTM-LSTM model is better than LSTM-CRF model based on our tagging scheme. Because, LSTM is capable of learning long-term dependencies and CRF (Lafferty et al., 2001) is good at capturing the joint probability of the entire sequence of labels. The related tags may have a long distance from each other. Hence, LSTM decoding manner is a little better than CRF. LSTM-LSTM-Bias adds a bias weight to enhance the effect of entity tags and weaken the effect of invalid tag. Therefore, in this tagging scheme, our method can be better than the common LSTM-decoding methods.
5 Analysis and Discussion
5.1 Error Analysis
In this paper, we focus on extracting triplets composed of two entities and a relation. Table 1 has shown the predict results of the task. It treats an triplet is correct only when the relation type and the head offsets of two corresponding entities are both correct. In order to find out the factors that affect the results of end-to-end models, we analyze the performance on predicting each element in the triplet as Table 2 shows. E1 and E2 represent the performance on predicting each entity, respectively. If the head offset of the first entity is correct, then the instance of E1 is correct, the same to E2. Regardless of relation type, if the head offsets of two corresponding entities are both correct, the instance of (E1, E2) is correct.
As shown in Table 2, (E1, E2) has higher precision when compared with E1 and E2. But its recall result is lower than E1 and E2. It means that some of the predicted entities do not form a pair. They only obtain E1 and do not find its corresponding E2, or obtain E2 and do not find its corresponding E1. Thus it leads to the prediction of more single E and less (E1, E2) pairs. Therefore, entity pair (E1, E2) has higher precision and lower recall than single E. Besides, the predict-
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
691
692
693
694
695
696
697
698
699
ed results of (E1, E2) in Table 2 have about 3% improvement when compared predicted results in Table 1, which means that 3% of the test data is predicted to be wrong because the relation type is predicted to be wrong.
5.2 Analysis of Bias
Different from LSTM-CRF and LSTM-LSTM, our approach is biased towards relational labels to enhance links between entities. In order to further analyze the effect of the bias objective function, we visualize the ratio of predicted single entities for each end-to-end method as Figure 3. The single entities refer to those who cannot find their corresponding entities. Figure 3 shows whether it is E1 or E2, our method can get a relatively low ratio on the single entities. It means that our method can effectively associate two entities when compared LSTM-CRF and LSTM-LSTM which pay little attention to the relational tags.
Single E1 Single E2 0.00
0.05
0.10
0.15
0.20
0.25
T h e R
a ti
o o
f S in
g le
E
0.178 0.186
0.151 0.167
0.135
0.101
LSTM-CRF LSTM-LSTM LSTM-LSTM-Bias
Figure 3: The ratio of predicted single entities for each method. The higher of the ratio the more entities are left.
Besides, we also change the Bias Parameter α from 1 to 20, and the predicted results are shown in Figure 4. If α is too large, it will affect the accuracy of prediction and if α is too small, the recall will decline. When α = 10, LSTM-LSTMBias can balance the precision and recall, and can achieve the best F1 scores.
5.3 The Applicability of Our Methods
Although LSTM-LSTM-Bias can achieve a maximum F1 value, in practice we may pay more attention to the precision or recall. Because, LSTMLSTM and LSTM-LSTM-Bias use the softmax
function to select the maximum probability label as the predicted tag for each word. We reset the prediction method as follows: if the maximum predicted probability is smaller than the threshold, the predicted tag is ’O’, else the result is the maximum probability label. Therefore, we tune the threshold for LSTM-LSTM and LSTM-LSTMBias to obtain their Precision-Recall curves that can reflect the predictive ability of models. Figure 5 shows that LSTM-LSTM-Bias is larger than LSTM-LSTM in both precision and recall, which means a more broad applicability.
5.4 Case Study
In this section, we observe the prediction results of end-to-end methods, and then select several representative examples to illustrate the advantages and
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Standard S1 [Panama City Beach]E2contain has condos , but the area was one of only two in [Florida]E1contain where sales rose in March , compared with a year earlier.
LSTM-LSTM Panama City Beach has condos , but the area was one of only two in [Florida]E1contain where sales rose in March , compared with a year earlier.
LSTM-LSTM-Bias [Panama City Beach]E2contain has condos , but the area was one of only two in [Florida]E1contain where sales rose in March , compared with a year earlier.
Standard S2 All came from [Nuremberg]E2contain , [Germany]E1contain , a center of brass production since the Middle Ages. LSTM-LSTM All came from Nuremberg , [Germany]E1contain , a center of brass production since the [Middle Ages]E2contain. LSTM-LSTM-Bias All came from Nuremberg , [Germany]E1contain , a center of brass production since the [Middle Ages]E2contain.
Standard S3 [Stephen A.]E2CF , the co-founder of the [Blackstone Group]E1CF , which is in the process of going public , made $ 400 million last year. LSTM-LSTM [Stephen A.]E1CF , the co-founder of the [Blackstone Group]E1CF , which is in the process of going public , made $ 400 million last year. LSTM-LSTM-Bias [Stephen A.]E1CF , the co-founder of the [Blackstone Group]E2CF , which is in the process of going public , made $ 400 million last year.
Table 3: Output from different models. Standard Si represents the gold standard of sentence i. The blue part is the correct result, and the red one is the wrong one. E1CF in case ’3’ is short for E1Company−Founder.
disadvantages of the methods. Each example contains three row, the first row is the gold standard, the second and the third rows are the extracted results of model LSTM-LSTM and LSTM-LSTMBias respectively. Example S1 represents the situation that the distance between the two interrelated entities is far away from each other, which is more difficult to detect their relationships. When compared with LSTM-LSTM, LSTM-LSTM-Bias uses a bias objective function which enhance the relevance between entities. Therefore, in this example, LSTM-LSTM-Bias can extract two related entities, while LSTM-LSTM can only extract one entity of “Florida” and can not detect entity “Panama City Beach”.
S2 is a negative example that shows these methods may mistakenly predict one of the entity. There are no indicative words between entities Nuremberg and Germany. Besides, the patten “a * of *” between Germany and MiddleAges may be easy to mislead the models that there exists a relation of “Contains” between them. The problem can be solved by adding some samples of this kind of expression patterns to the training data.
S3 is a case that models can predict the entities’ head offset right, but the relational role is
wrong. LSTM-LSTM treats both “Stephen A. Schwarzman” and “Blackstone Group” as entity E1, and can not find its corresponding E2. Although, LSTM-LSMT–Bias can find the entities pair (E1, E2), it reverses the roles of “Stephen A. Schwarzman” and “Blackstone Group”. It shows that LSTM-LSTM-Bias is able to better on predicting entities pair, but it remains to be improved in distinguishing the relationship between the two entities.
6 Conclusion
In this paper, we propose a novel tagging scheme and investigate the end-to-end models to jointly extract entities and relations. The experimental results show the effectiveness of our proposed method. But it still has shortcoming on the identification of the overlapping relations. In the future work, we will replace the softmax function in the output layer with multiple classifier, so that a word can has multiple tags. In this way, a word can appear in multiple triplet results, which can solve the problem of overlapping relations. Although, our model can enhance the effect of entity tags, the association between two corresponding entities still requires refinement in next works.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths:  When introducing the task, the authors use illustrative examples as well as the contributions of this paper.  Related Works section covers the state of the art, at the same time pointing similarities and differences between related Works and the proposed method. The presentation of the method is very clear, since the authors separate the tagging scheme and the end-to-end model. Another strong point of this work is the baselines used to compare the proposed methods with several classical triplet extraction methods. At last, the presentation of examples from dataset used to illustrate the advantages and disadvantages of the methods was very important. These outputs complement the explanation of tagging and evaluation of triplets.   - Weaknesses:  One of the main contributions of this paper is a new tagging scheme described in Section 3.1, however there are already other schemes for NER and RE being used, such as IO, BIO and BILOU.  Did the authors perform any experiment using other tagging scheme for this method? Regarding the dataset, in line 14, page 5, the authors cite the number of relations (24), but they do not mention the number or the type of named entities. In Section 4.1, the evaluation criteria of triplets are presented. These criteria were based on previous work? As I see it, the stage of entity identification is not complete if you consider only the head of the entity. Regarding example S3, shown in Table 3, the output of the LSTM-LSTM-Bias was considered correct? The text states that the relation role is wrong, although it is not clear if the relation role is considered in the evaluation.   - General Discussion:  This paper proposes a novel tagging scheme and investigates the end-to-end models to jointly extract entities and relations.  The article is organized in a clear way and it is well written, which makes it easy to understand the proposed method.