An Interpretable Knowledge Transfer Model for Knowledge Base Completion
Knowledge bases are important resources for a variety of natural language processing tasks but suffer from incompleteness. We propose a novel embedding model, ITransF, to perform knowledge base completion. Equipped with a sparse attention mechanism, ITransF discovers hidden concepts of relations and transfer statistical strength through the sharing of concepts. Moreover, the learned associations between relations and concepts, which are represented by sparse attention vectors, can be interpreted easily. We evaluate ITransF on two benchmark datasets— WN18 and FB15k for knowledge base completion and obtains improvements on both the mean rank and Hits@10 metrics, over all baselines that do not use additional information.
1 Introduction
Knowledge bases (KB), such as WordNet (Fellbaum, 1998), Freebase (Bollacker et al., 2008), YAGO (Suchanek et al., 2007) and DBpedia (Lehmann et al., 2015), are useful resources for many applications such as question answering (Berant et al., 2013; Yih et al., 2015; Dai et al., 2016) and information extraction (Mintz et al., 2009). However, knowledge bases suffer from incompleteness despite their formidable sizes (Socher et al., 2013; West et al., 2014), leading to a number of studies on automatic knowledge base completion (KBC) (Nickel et al., 2015) or link prediction.
The fundamental motivation behind these studies is that there exist some statistical regularities under the intertwined facts stored in the multirelational knowledge base. By discovering gener-
alizable regularities in known facts, missing ones may be recovered in a faithful way. Due to its excellent generalization capability, distributed representations, a.k.a. embeddings, have been popularized to address the KBC task (Nickel et al., 2011; Bordes et al., 2011, 2014, 2013; Socher et al., 2013; Wang et al., 2014; Guu et al., 2015; Nguyen et al., 2016b).
As a seminal work, Bordes et al. (2013) proposes the TransE, which models the statistical regularities with linear translations between entity embeddings operated by a relation embedding. Implicitly, TransE assumes both entity embeddings and relation embeddings dwell in the same vector space, posing an unnecessarily strong prior. To relax this requirement, a variety of models first project the entity embeddings to a relationdependent space (Bordes et al., 2014; Ji et al., 2015; Lin et al., 2015b; Nguyen et al., 2016b), and then model the translation property in the projected space. Typically, these relation-dependent spaces are characterized by the projection matrices unique to each relation. As a benefit, different aspects of the same entity can be temporarily emphasized or depressed as an effect of the projection. For instance, STransE (Nguyen et al., 2016b) utilizes two projection matrices per relation, one for the head entity and the other for the tail entity.
Despite the superior performance of STransE compared to TransE, it is more prone to the data sparsity problem. Concretely, since the projection spaces are unique to each relation, projection matrices associated with rare relations can only be exposed to very few facts during training, resulting in poor generalization. For common relations, a similar issue exists. Without any restrictions on the number of projection matrices, logically related or conceptually similar relations may have distinct projection spaces, hindering the discovery, sharing, and generalization of statistical regularities.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Previously, a line of research makes use of external information such as textual relations from web-scale corpus or node features (Toutanova et al., 2015; Toutanova and Chen, 2015; Nguyen et al., 2016a), alleviating the sparsity problem. In parallel, recent work has proposed to model regularities beyond local facts by considering multirelation paths (Garcı́a-Durán et al., 2015; Lin et al., 2015a; Shen et al., 2016). Since the number of paths grows exponentially with its length, as a side effect, path-based models enjoy much more training cases, suffering less from the problem.
In this paper, we propose an interpretable knowledge transfer model (ITransF), which encourages the sharing of statistic regularities between the projection matrices of relations and alleviates the data sparsity problem. At the core of ITransF is a sparse attention mechanism, which learns to compose shared concept matrices into relation-specific projection matrices, leading to a better generalization property. Without any external resources, ITransF improves mean rank and Hits@10 on two benchmark datasets, over all previous approaches of the same kind. In addition, the parameter sharing is clearly indicated by the learned sparse attention vectors, enabling us to interpret how knowledge transfer is carried out. To induce the desired sparsity during optimization, we further introduce a block iterative optimization algorithm.
In summary, the contributions of this work are: (i) proposing a novel knowledge embedding model which enables knowledge transfer by learning to discover shared regularities; (ii) introducing a learning algorithm to directly optimize a sparse representation from which the knowledge transferring procedure is interpretable; (iii) showing the effectiveness of our model by outperforming baselines on two benchmark datasets for knowledge base completion task.
2 Notation and Previous Models
Let E denote the set of entities and R denote the set of relations. In knowledge base completion, given a training set P of triples (h, r, t) where h, t 2 E are the head and tail entities having a relation r 2 R, e.g., (Steve Jobs, FounderOf, Apple), we want to predict missing facts such as (Steve Jobs, Profession, Businessperson).
Most of the embedding models for knowledge base completion define an energy function fr(h, t)
according to the fact’s plausibility (Bordes et al., 2011, 2014, 2013; Socher et al., 2013; Wang et al., 2014; Yang et al., 2015; Guu et al., 2015; Nguyen et al., 2016b). The models are learned to minimize energy fr(h, t) of a plausible triple (h, r, t) and to maximize energy fr(h0, t0) of an implausible triple (h0, r, t0).
Motivated by the linear translation phenomenon observed in well trained word embeddings (Mikolov et al., 2013), TransE (Bordes et al., 2013) represents the head entity h, the relation r and the tail entity t with vectors h, r and t 2 Rn respectively, which were trained so that h+r ⇡ t. They define the energy function as
fr(h, t) = kh+ r tk`
where ` = 1 or 2, which means either the `1 or the `2 norm of the vector h + r t will be used depending on the performance on the validation set.
To better model relation-specific aspects of the same entity, TransR (Lin et al., 2015b) uses projection matrices and projects the head entity and the tail entity to a relation-dependent space. STransE (Nguyen et al., 2016b) extends TransR by employing different matrices for mapping the head and the tail entity. The energy function is
fr(h, t) = kWr,1h+ r Wr,2tk`
However, not all relations have abundant data to estimate the relation specific matrices as most of the training samples are associated with only a few relations, leading to the data sparsity problem for rare relations.
3 Interpretable Knowledge Transfer
3.1 Model
As discussed above, a fundamental weakness in TransR and STransE is that they equip each relation with a set of unique projection matrices, which not only introduces more parameters but also hinders knowledge sharing. Intuitively, many relations share some concepts with each other, although they are stored as independent symbols in KB. For example, the relation “(somebody) won award for (some work)” and “(somebody) was nominated for (some work)” both describe a person’s high-quality work which wins an award or a nomination respectively. This phenomenon suggests that one relation actually represents a collection of real-world concepts, and one concept
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
can be shared by several relations. Inspired by the existence of such lower-level concepts, instead of defining a unique set of projection matrices for every relation, we can alternatively define a small set of concept projection matrices and then compose them into customized projection matrices. Effectively, the relation-dependent translation space is then reduced to the smaller concept spaces.
However, in general, we do not have prior knowledge about what concepts exist out there and how they are composed to form relations. Therefore, in ITransF, we propose to learn this information simultaneously from data, together with all knowledge embeddings. Following this idea, we first present the model details, then discuss the optimization techniques for training.
Energy function Specifically, we stack all the concept projection matrices to a 3-dimensional tensor D 2 Rm⇥n⇥n, where m is the pre-specified number of concept projection matrices and n is the dimensionality of entity embeddings and relation embeddings. We let each relation select the most useful projection matrices from the tensor, where the selection is represented by an attention vector. The energy function of ITransF is defined as:
fr(h, t) = k↵Hr ·D · h+ r ↵Tr ·D · tk` (1)
where ↵Hr ,↵Tr 2 [0, 1]m, satisfying P
i↵ H r,i =P
i↵ T r,i = 1, are normalized attention vectors used to compose all concept projection matrices in D by a convex combination. It is obvious that STransE can be expressed as a special case of our model when we use m = 2|R| concept matrices and set attention vectors to disjoint one-hot vectors. Hence our model space is a generalization of STransE. Note that we can safely use fewer concept matrices in ITransF and obtain better performance (see section 4.3), though STransE always requires 2|R| projection matrices.
We follow previous work to minimize the following hinge loss function:
L = X
(h,r,t)⇠P, (h0,r,t0)⇠N
⇥ + fr(h, t) fr(h0, t0) ⇤ + (2)
where P is the training set consisting of correct triples, N is the distribution of corrupted triples defined in section 3.3, and [·]+ = max(·, 0). Note that we have omitted the dependence of N on (h, r, t) to avoid clutter. We normalize the entity vectors h, t, and the projected entity vectors
↵Hr ·D · h and ↵Tr ·D · t to have unit length after each update, which is an effective regularization method that benefits all models.
Sparse attention vectors In Eq. (1), we have defined ↵Hr ,↵Tr to be some normalized vectors used for composition. With a dense attention vector, it is computationally expensive to perform the convex combination of m matrices in each iteration. Moreover, a relation usually does not consist of all existing concepts in practice. Furthermore, when the attention vectors are sparse, it is often easier to interpret their behaviors and understand how concepts are shared by different relations.
Motivated by these potential benefits, we further hope to learn sparse attention vectors in ITransF. However, directly posing `1 regularization (Tibshirani, 1996) on the attention vectors fails to produce sparse representations in our preliminary experiment, which motivates us to enforce `0 constraints on ↵Tr ,↵Hr .
In order to satisfy both the normalization condition and the `0 constraints, we reparameterize the attention vectors in the following way:
↵Hr = SparseSoftmax(v H r , I H r ) ↵Tr = SparseSoftmax(v T r , I T r )
where vHr ,vTr 2 Rm are the pre-softmax scores, IHr , I T r 2 {0, 1}m are the sparse assignment vectors, indicating the non-zero entries of attention vectors, and the SparseSoftmax is defined as
SparseSoftmax(v, I)i = exp(vi/⌧)IiP j exp(vj/⌧)Ij
with ⌧ being the temperature of Softmax. With this reparameterization, vHr ,vTr and IHr , I T r replace ↵Tr ,↵Hr to become the real parameters of the model. Also, note that it is equivalent to pose the `0 constraints on IHr , ITr instead of ↵Tr ,↵ H r . Putting these modifications together, we can rewrite the optimization problem as
minimize L subject to kIHr k0  k, kITr k0  k
(3)
where L is the loss function defined in Eq. (2).
3.2 Block Iterative Optimization
Though sparseness is favorable in practice, it is generally NP-hard to find the optimal solution under `0 constraints. Thus, we resort to an approximated algorithm in this work.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
For convenience, we refer to the parameters with and without the sparse constraints as the sparse partition and the dense partition, respectively. Based on this notion, the high-level idea of the approximated algorithm is to iteratively optimize one of the two partitions while holding the other one fixed. Since all parameters in the dense partition, including the embeddings, the projection matrices, and the pre-softmax scores, are fully differentiable with the sparse partition fixed, we can simply utilize SGD to optimize the dense partition. Then, the core difficulty lies in the step of optimizing the sparse partition (i.e. the sparse assignment vectors), during which we want the following two properties to hold
1. the sparsity required by the `0 constaint is maintained, and
2. the cost define by Eq. (2) is decreased.
Satisfying the two criterion seems to highly resemble the original problem defined in Eq. (3). However, the dramatic difference here is that with parameters in the dense partition regarded as constant, the cost function is decoupled w.r.t. each relation r. In other words, the optimal choice of IHr , I T r is independent of IHr0 , I T r0 for any r
0 6= r. Therefore, we only need to consider the optimization for a single relation r, which is essentially an assignment problem. Note that, however, IHr and ITr are still coupled, without which we basically reach the situation in a backpack problem. In principle, one can explore combinatorial optimization techniques to optimize IHr0 , I T r0 jointly, which usually involve some iterative procedure. To avoid adding another inner loop to our algorithm, we turn to a simple but fast approximation method based on the following single-matrix cost.
Specifically, for each relation r, we consider the induced cost LHr,i where only a single projection matrix i is used for the head entity:
LHr,i = X
(h,r,t)⇠Pr, (h0,r,t0)⇠Nr
⇥ + fHr,i(h, t) fHr,i(h0, t0) ⇤ +
where fHr,i(h, t) = kDi · h + r ↵Tr · D · tk is the corresponding energy function, and the subscript in Pr and Nr denotes the subsets with relation r. Intuitively, LHr,i measures, given the current tail attention vector ↵Tr , if only one project matrix could be chosen for the head entity, how implausible Di would be. Hence, i⇤ = argmini LHr,i gives
us the best single projection matrix on the head side given ↵Tr .
Now, in order to choose the best k matrices, we basically ignore the interaction among projection matrices, and update IHr in the following way:
IHr,i ( 1, i 2 argpartitioni(LHr,i, k) 0, otherwise
where the function argpartitioni(xi, k) produces the index set of the lowest-k values of xi.
Analogously, we can define the single-matrix cost LTr,i and the energy function fTr,i(h, t) on the tail side in a symmetric way. Then, the update rule for IHr follows the same derivation. Admittedly, the approximation described here is relatively crude. But as we will show in section 4, the proposed algorithm yields good performance empirically. We leave the further improvement of the optimization method as future work.
3.3 Corrupted Sample Generating Method
Recall that we need to sample a negative triple (h0, r, t0) to compute hinge loss shown in Eq. 2, given a positive triple (h, r, t) 2 P . The distribution of negative triple is denoted by N(h, r, t). Previous work (Bordes et al., 2013; Lin et al., 2015b; Yang et al., 2015; Nguyen et al., 2016b) generally constructs a set of corrupted triples by replacing the head entity or tail entity with a random entity uniformly sampled from the KB.
However, uniformly sampling corrupted entities may not be optimal. Often, the head and tail entities associated a relation can only belong to a specific domain. When the corrupted entity comes from other domains, it is very easy for the model to induce a large energy gap between true triple and corrupted one. As the energy gap exceeds , there will be no training signal from this corrupted triple. In comparison, if the corrupted entity comes from the same domain, the task becomes harder for the model, leading to more consistent training signal.
Motivated by this observation, we propose to sample corrupted head or tail from entities in the same domain with a probability pr and from the whole entity set with probability 1 pr. The choice of relation-dependent probability pr is specified in Appendix A.1. In the rest of the paper, we refer to the new proposed sampling method as ”domain sampling”.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
4 Experiments
4.1 Setup
To evaluate link prediction, we conduct experiments on the WN18 (WordNet) and FB15k (Freebase) introduced by Bordes et al. (2013) and use the same training/validation/test split as in (Bordes et al., 2013). The information of the two datasets is given in Table 1.
Dataset #E #R #Train #Valid #Test WN18 40,943 18 141,442 5,000 5,000 FB15k 14,951 1,345 483,142 50,000 59,071
Table 1: Statistics of FB15k and WN18 used in experiments. #E, #R denote the number of entities and relation types respectively. #Train, #Valid and #Test are the numbers of triples in the training, validation and test sets respectively.
In knowledge base completion task, we evaluate model’s performance of predicting the head entity or the tail entity given the relation and the other entity. For example, to predict head given relation r and tail t in triple (h, r, t), we compute the energy function fr(h0, t) for each entity h0 in the knowledge base and rank all the entities according to the energy. We follow Bordes et al. (2013) to report the filter results, i.e., removing all other correct candidates h0 in ranking. The rank of the correct entity is then obtained and we report the mean rank (mean of the predicted ranks) and Hits@10 (top 10 accuracy). Lower mean rank or higher Hits@10 mean better performance.
4.2 Implementation Details
We initialize the projection matrices with identity matrices added with a small noise sampled from normal distribution N (0, 0.0052). The entity and relation vectors of ITransF are initialized by TransE (Bordes et al., 2013), following Lin et al. (2015b); Ji et al. (2015); Garcı́a-Durán et al. (2016, 2015); Lin et al. (2015a). We ran minibatch SGD until convergence. We employ the “Bernoulli” sampling method to generate incorrect triples as used in Wang et al. (2014), Lin et al. (2015b), He et al. (2015), Ji et al. (2015) and Lin et al. (2015a).
STransE (Nguyen et al., 2016b) is the most similar knowledge embedding model to ours except that they use distinct projection matrices for each relation. We use the same hyperparameters as used in STransE and no significant improvement is observed when we alter hyperparameters. We set the
margin to 5 and dimension of embedding n to 50 for WN18, and = 1, n = 100 for FB15k. We set the batch size to 20 for WN18 and 1000 for FB15k. The learning rate is 0.01 on WN18 and 0.1 on FB15k. We use 30 matrices on WN18 and 300 matrices on FB15k. All the models are implemented with Theano (Bergstra et al., 2010). The Softmax temperature is set to 1/4.
4.3 Results & Analysis
The overall link prediction results1 are reported in Table 2. Our model consistently outperforms previous models without external information on both the metrics of WN18 and FB15k. On WN18, we even achieve a much better mean rank with comparable Hits@10 than current state-of-the-art model IRN employing external information.
We can see that path information is very helpful on FB15k and models taking advantage of path information outperform intrinsic models by a significant margin. Indeed, a lot of facts are easier to recover with the help of multi-step inference. For example, if we know Barack Obama is born in Honolulu, a city in the United States, then we easily know the nationality of Obama is the United States. An straightforward way of extending our proposed model to k-step path P = {ri}ki=1 is to define a path energy function k↵HP · D · h +P
ri2P ri ↵ T P · D · tk`, ↵HP is a concept association related to the path. We plan to extend our model to multi-step path in the future.
To provide a detailed understanding why the proposed model achieves better performance, we present some further analysis in the sequel.
Performance on Rare Relations In the proposed ITransF, we design an attention mechanism to encourage knowledge sharing across different relations. Naturally, facts associated with rare relations should benefit most from such sharing, boosting the overall performance. To verify this hypothesis, we investigate our model’s performance on relations with different frequency. Specifically, we sort all relations by their frequency in the training set, and split them into 5 buckets evenly. Within each bucket, we compare our model with STransE, as shown in Fig-
1Note that although IRN (Shen et al., 2016) does not explicitly exploit path information, it performs multi-step inference through the multiple usages of external memory. When IRN is allowed to access memory once for each prediction, its Hits@10 is 80.7, similar to models without path information.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Model Additional Information WN18 FB15kMean Rank Hits@10 Mean Rank Hits@10 SE (Bordes et al., 2011) No 985 80.5 162 39.8 Unstructured (Bordes et al., 2014) No 304 38.2 979 6.3 TransE (Bordes et al., 2013) No 251 89.2 125 47.1 TransH (Wang et al., 2014) No 303 86.7 87 64.4 TransR (Lin et al., 2015b) No 225 92.0 77 68.7 CTransR (Lin et al., 2015b) No 218 92.3 75 70.2 KG2E (He et al., 2015) No 348 93.2 59 74.0 TransD (Ji et al., 2015) No 212 92.2 91 77.3 TATEC (Garcı́a-Durán et al., 2016) No - - 58 76.7 NTN (Socher et al., 2013) No - 66.1 - 41.4 DISTMULT (Yang et al., 2015) No - 94.2 - 57.7 STransE (Nguyen et al., 2016b) No 206 (244) 93.4 (94.7) 69 79.7 ITransF No 205 94.2 65 81.0 ITransF (domain sampling) No 223 95.2 77 81.4 RTransE (Garcı́a-Durán et al., 2015) Path - - 50 76.2 PTransE (Lin et al., 2015a) Path - - 58 84.6 NLFeat (Toutanova and Chen, 2015) Node + Link Features - 94.3 - 87.0 Random Walk (Wei et al., 2016) Path - 94.8 - 74.7 IRN (Shen et al., 2016) External Memory 249 95.3 38 92.7
Table 2: Link prediction results on two datasets. Higher Hits@10 or lower Mean Rank indicates better performance. Following Nguyen et al. (2016b) and Shen et al. (2016), we divide the models into two groups. The first group contains intrinsic models without using extra information. The second group make use of additional information. Results in the brackets are another set of results STransE reported.
ure 1.2 As we can see, on WN18, ITransF outperforms STransE by a significant margin on rare relations. In particular, in the last bin (rarest relations), the average Hits@10 increases from 55.2 to 93.8, showing the great benefits of transferring statistical strength from common relations to rare ones. On FB15k, we can also observe a similar pattern, although the degree of improvement is less significant. We conjecture the difference roots in the fact that many rare relations on FB15k have disjoint domains, knowledge transfer through common concepts is harder.
Interpretability In addition to the quantitative evidence supporting the effectiveness of knowledge sharing, we provide some intuitive examples to show how knowledge is shared in our model. As we mentioned earlier, the sparse attention vectors fully capture the association between relations and concepts and hence the knowledge transfer among relations. Thus, we visualize the attention vectors for several relations on both WN18 and FB15K in Figure 2.
For WN18, the words “hyponym” and “hypernym” refer to words with more specific or general meaning respectively. For example, PhD is a hyponym of student and student is a hypernym of PhD. As we can see, concepts associated with the head entities in one relation are also associated
2Domain sampling is not employed.
with the tail entities in its reverse relation. Further, “instance hypernym” is a special hypernym with the head entity being an instance, and the tail entity being an abstract notion. A typical example is (New York,instance hypernym, city). This connection has also been discovered by our model, indicated by the fact that “instance hypernym(T)” and “hypernym(T)” share a common concept matrix. Finally, for symmetric relations like “similar to”, we see the head attention is identical to the tail attention, which well matches our intuition.
On FB15k, we also see the sharing between reverse relations, as in “(somebody) won award for (some work)” and “(some work) award winning work (somebody)”. What’s more, although relation “won award for” and “was nominated for” share the same concepts, their attention distributions are different, suggesting distinct emphasis. Finally, symmetric relations like spouse behave similarly as mentioned before.
Model Compression A byproduct of parameter sharing mechanism employed by ITransF is a much more compact model with equal performance. Figure 3 plots the average performance of ITransF against the number of projection matrices m, together with two baseline models. On FB15k, when we reduce the number of matrices from 2200 to 30 (⇠ 90⇥ compression), our model performance decreases by only 0.09% on
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
674
675
676
677
678
679
680
681
687
688
689
690
691
692
693
694
695
696
697
698
699
H its
@ 10
0
25
50
75
100
Relation Bin 1 2 3 4 5
ITransF STransE
(a) WN18
H its
@ 10
0
25
50
75
100
Relation Bin 1 2 3 4 5
ITransF STransE
(b) FB15k
(a) WN18 (b) FB15k
Figure 2: Heatmap visualization of attention vectors on WN18 and FB15k. Each row is an attention vector ↵Hr or ↵Tr for a relation’s head or tail concepts.
Hits@10, still outperforming STransE. Similarly, on WN18, ITransF continues to achieve the best performance when we reduce the number of concept project matrices to 18.
5 Analysis on Sparseness
Sparseness is desirable since it contribute to interpretability and computational efficiency of our model. We investigate whether enforcing sparseness would deteriorate the model performance and compare our method with another sparse encoding methods in this section.
Dense Attention vs Sparse Attention Although sparsity usually enjoys many practical advantages, it may deteriorate the model performance when applied improperly. Here, we show that our model employing sparse attention can achieve similar results with dense attention with a significantly less computational burden. We compare models in a setting where the computation time of dense attention model is acceptable3. We use 22 weight
3With 300 projection matrices, it takes 1h1m to run one epoch for a model with dense attention.
matrices on WN18 and 15 weight matrices on FB15k and train both the models for 1500 epochs. The results are reported in table 3. Generally, ITransF with sparse attention has slightly better performance comparing to dense attention except a higher mean rank on WN18.
Nonnegative Sparse Encoding In the proposed model, we induce the sparsity by a carefully designed iterative optimization procedure. Apart from this approach, one may utilize sparse encoding techniques to obtain sparseness based on the pretrained projection matrices from STransE. Concretely, stacking |2R| pretrained projection matrices into a 3-dimensional tensor X 2
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
H its
@ 10
70
73.25
76.5
79.75
83
# matrices 15 30 75 300 600 1200 1345 2200 2690
ITransF STransE CTransR
(a) FB15k
H its
@ 10
90
91.25
92.5
93.75
95
# matrices 18 22 26 30 36 45
ITransF STransE CTransR
(b) WN18
Figure 3: Performance with different number of projection matrices. Note that the X-axis denoting the number of matrices is not linearly scaled.
R2|R|⇥n⇥n, similar sparsity can be induced by solving an `1-regularized tensor completion problem minA,D ||X DA||22 + kAk`1 . Basically, A plays the same role as the attention vectors in our model. For more details, we refer readers to (Faruqui et al., 2015).
For completeness, we compare our model with the aforementioned approach4. The comparison is summarized in table 4. On both benchmarks, ITransF achieves significant improvement against sparse encoding on pretrained model. This performance gap should be expected since the objective function of sparse encoding methods is to minimize the reconstruction loss rather than optimize the criterion for link prediction.
Method WN18 FB15kMR H10 MR H10 Sparse Encoding 211 86.6 66 79.1 ITransF 205 94.2 65 81.0
Table 4: Different methods to obtain sparse representations
6 Related Work
In KBC, CTransR (Lin et al., 2015b) enables relation embedding sharing across similar relations, but they cluster relations before training rather than learning it in a principled way. Further, they do not solve the data sparsity problem because there is no sharing of projection matrices which have a lot more parameters.
Data sparsity is a common problem in many fields. Transfer learning (Pan and Yang, 2010) has been shown to be promising to transfer knowledge and statistical strengths across similar models or languages. For example, Bharadwaj et al.
4We use the toolkit provided by (Faruqui et al., 2015).
(2016) transfers models on resource-rich languages to low resource languages by parameter sharing through common phonological features in name entity recognition. Zoph et al. (2016) initialize from models trained by resource-rich languages to translate low-resource languages.
Several works on obtaining a sparse attention (Martins and Astudillo, 2016; Makhzani and Frey, 2014; Shazeer et al., 2017) share a similar idea of sorting the values before softmax and only keeping the K largest values. However, the sorting operation in these works is not GPU-friendly.
The block iterative optimization algorithm in our work is inspired by LightRNN (Li et al., 2016). They allocate every word in the vocabulary in a table. A word is represented by a row vector and a column vector depending on its position in the table. They iteratively optimize embeddings and allocation of words in tables.
7 Conclusion and Future Work
In summary, we propose a knowledge embedding model which can discover shared hidden concepts, and design a learning algorithm to induce the interpretable sparse representation. Empirically, we show our model can improve the performance on two benchmark datasets without external resources, over all previous models of the same kind.
In the future, we plan to enable ITransF to perform multi-step inference, and extend the sharing mechanism to entity and relation embeddings, further enhancing the statistical binding across parameters. In addition, our framework can also be applied to multi-task learning, promoting a finer sharing among different tasks.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This paper considers the problem of KB completion and proposes ITransF for this purpose. Unlike STransE that assigns each relation an independent matrix, this paper proposes to share the parameters between different relations. A model is proposed where a tensor D is constructed that contains various relational matrices as its slices and a selectional vector   alpha is used to select a subset of relevant relational matrix for composing a particular semantic relation. The paper then discuss a technique to make   alpha sparse. Experimental results on two standard benchmark datasets shows the superiority of ITransF over prior proposals.  The paper is overall well written and the experimental results are good. However, I have several concerns regarding this work that I hope the authors will answer in their response.  1. Just by arranging relational matrices in a tensor and selecting (or more appropriately considering a linearly weighted sum of the relational matrices) does not ensure any information sharing between different relational matrices. This would have been the case if you had performed some of a tensor decomposition and projected the different slices (relational matrices) into some common lower-dimensional core tensor. It is not clear why this approach was not taken despite the motivation to share information between different relational matrices. 2. The two requirements (a) to share information across different relational matrices and (b) make the attention vectors sparse are some what contradictory. If the attention vector is truly sparse and has many zeros then information will not flow into those slices during optimisation.  3. The authors spend a lot of space discussing techniques for computing sparse attention vectors. The authors mention in page 3 that   ell_1 regularisation did not work in their preliminary experiments. However, no experimental results are shown for   ell_1 regularisation nor they explain why   ell_1 is not suitable for this task. To this reviewer, it appears as an obvious baseline to try, especially given the ease of optimisation. You use   ell_0 instead and get into NP hard optimisations because of it. Then you propose a technique and a rather crude approximation in the end to solve it. All that trouble could be spared if   ell_1 was used. 4. The vector   alpha is performing a selection or a weighing over the slices of D. It is slightly misleading to call this as  attention d as it is a term used in NLP for a more different type of models (see attention model used in machine translation). 5. It is not clear why you need to initialise the optimisation by pre-trained embeddings from TransE. Why cannot you simply randomly initialise the embeddings as done in TransE and then update them? It is not fair to compare against TransE if you use TransE as your initial point.  Learning the association between semantic relations is an idea that has been used in related problems in NLP such as relational similarity measurement [Turney JAIR 2012] and relation adaptation [Bollegala et al. IJCAI 2011]. It would be good to put the current work with respect to such prior proposals in NLP for modelling inter-relational correlation similarity.  Thanks for providing feedback. I have read it.


Neural word segmentation research has benefited from large-scale raw texts by leveraging them for pretraining character and word embeddings. On the other hand, statistical segmentation research has exploited richer sources of external information, such as punctuation, automatic segmentation and POS. We investigate the effectiveness of a range of external training sources for neural word segmentation by building a modular segmentation model, pretraining the most important sub module using rich external sources. Results show that such pretraining significantly improves the model, leading to accuracies competitive to the best methods on six benchmarks.
1 Introduction
There has been a recent shift of research attention in the word segmentation literature from statistical methods to deep learning (Zheng et al., 2013; Pei et al., 2014; Morita et al., 2015; Chen et al., 2015b; Cai and Zhao, 2016; Zhang et al., 2016). Neural network models have been exploited due to their strength in non-sparse representation learning and non-linear power in feature combination, which have led to advances in many NLP tasks. So far, neural word segmentors have given comparable accuracies to the best statictical models.
With respect to non-sparse representation, character embeddings have been exploited as a foundation of neural word segmentors. They serve to reduce sparsity of character ngrams, allowing, for example, “猫(cat)躺(lie)在(in)墙角(corner)” to be connected with “狗(dog) 蹲(sit) 在(in) 墙 角(corner)” (Zheng et al., 2013), which is infeasible by using sparse one-hot character features. In
addition to character embeddings, distributed representations of character bigrams (Mansur et al., 2013; Pei et al., 2014) and words (Morita et al., 2015; Zhang et al., 2016) have also been shown to improve segmentation accuracies.
With respect to non-linear modeling power, various network structures have been exploited to represent contexts for segmentation disambiguation, including multi-layer perceptrons on fivecharacter windows (Zheng et al., 2013; Mansur et al., 2013; Pei et al., 2014; Chen et al., 2015a), as well as LSTMs on characters (Chen et al., 2015b; Xu and Sun, 2016) and words (Morita et al., 2015; Cai and Zhao, 2016; Zhang et al., 2016). For structured learning and inference, CRF has been used for character sequence labelling models (Pei et al., 2014; Chen et al., 2015b) and structural beam search has been used for word-based segmentors (Cai and Zhao, 2016; Zhang et al., 2016).
Previous research has shown that segmentation accuracies can be improved by pretraining character and word embeddings over large Chinese texts, which is consistent with findings on other NLP tasks, such as parsing (Andor et al., 2016). Pretraining can be regarded as one way of leveraging external resources to improve accuracies, which is practically highly useful and has become a standard practice in neural NLP. On the other hand, statistical segmentation research has exploited raw texts for semi-supervised learning, by collecting clues from raw texts more thoroughly such as mutual information and punctuation (Li and Sun, 2009; Sun and Xu, 2011), and making use of selfpredictions (Wang et al., 2011; Liu and Zhang, 2012). It has also utilised heterogenous annotations such as POS (Ng and Low, 2004; Zhang and Clark, 2008) and segmentation under different standards (Jiang et al., 2009). To our knowledge, such rich external information has not been systematically investigated for neural segmentation.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
S A
hidden layer
output
车 站 那 边
w-k
Recognized words Partial word Incoming chars
w-2
我 之前 去 过 火
w-1 P c0 c1 . . . cm
. . . . . . . . .
. . .
XW XP XC
h
Figure 1: Overall model.
We fill this gap by investigating rich external pretraining for neural segmentation. Following Cai and Zhao (2016) and Zhang et al. (2016), we adopt a globally optimised beam-search framework for neural structured prediction (Andor et al., 2016; Zhou et al., 2015; Wiseman and Rush, 2016), which allows word information to be modelled explicitly. Different from previous work, we make our model conceptually simple and modular, so that the most important sub module, namely a five-character window context, can be pretrained using external data. We adopt a multi-task learning strategy (Collobert et al., 2011), casting each external source of information as a auxiliary classification task, sharing a five-character window network. After pretraining, the character window network is used to initialize the corresponding module in our segmentor.
Results on 6 different benchmarks show that our method outperforms the best statistical and neural segmentation models consistently, giving the best reported results on 5 datasets in different domains and genres. Our code and models can be downloaded from GitHub.com/XXXXXXXXX
2 Related Work
Work on statistical word segmentation dates back to the 1990s (Sproat et al., 1996). State-of-the-art approaches include character sequence labeling models (Xue et al., 2003) using CRFs (Peng et al., 2004; Zhao et al., 2006) and max-margin structured models leveraging word features (Zhang and Clark, 2007; Sun et al., 2009; Sun, 2010). Semisupervised methods have been applied to both character-based and word-based models, exploring external training data for better segmentation (Sun and Xu, 2011; Wang et al., 2011; Liu and
Zhang, 2012; Zhang et al., 2013). Our work belongs to recent neural word segmentation.
To our knowledge, there has no work in the literature systematically investigating rich external resources for neural word segmentation training. Closest in spirit to our work, Sun and Xu (2011) empirically studied the use of various external resources for enhancing a statistical segmentor, including character mutual information, access variety information, punctuation and other statistical information. Their baseline is similar to ours in the sense that both character and word contexts are considered. On the other hand, their model is statistical while ours is neural. Consequently, they integrate external knowledge as features, while we integrate it by shared network parameters. Our results show a similar degree of error reduction compared to theirs by using external data.
Our model inherits from previous findings on context representations, such as character windows (Mansur et al., 2013; Pei et al., 2014; Chen et al., 2015a) and LSTMs (Chen et al., 2015b; Xu and Sun, 2016). Similar to Zhang et al. (2016) and Cai and Zhao (2016), we use word context on top of character context. However, words play a relatively less important role in our model, and we find that word LSTM, which has been used by all previous neural segmentation work, is unnecessary for our model. Our model is conceptually simpler and more modularised compared with Zhang et al. (2016) and Cai and Zhao (2016), allowing a central sub module, namely a five-character context window, to be pretrained.
3 Model
Our segmentor works incrementally from left to right. At each step, the state consists of a sequence of words that have been fully recognized, denoted asW = [w−k, w−k+1, ..., w−1], a current partially recognized word P , and a sequence of next incoming characters, denoted as C = [c0, c1, ..., cm], as shown in Figure 1. Given an input sentence, W and P are initialized to [ ] and φ, respectively, and C contains all the input characters. At each step, a decision is made on c0, either appending it as a part of P , or seperating it as the beginning of a new word. The incremental process repeats until C is empty and P = φ again. Formally, the process can be regarded as a state-transition process, where a state is a tuple S = 〈W,P,C〉, and the transition actions include SEP and APP, as shown
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Axiom: S = 〈[ ], φ, C〉, V = 0 Goal: S = 〈W,φ, [ ]〉, V = Vfinal
SEP: S = 〈W,P, c0|C〉, V S ′ = 〈W |P, c0, C〉, V + Score(S, SEP)
APP: S = 〈W,P, c0|C〉, V S ′ = 〈W,P ⊕ c0, C〉, V + Score(S,APP)
Figure 2: Deduction system, where ⊕ denotes string concatenation.
by the deduction system in Figure 21. In the figure, V denotes the score of a state, given by a neural network model. The score of the initial state (i.e. axiom) is 0, and the score of a non-axiom state is the sum of scores of all incremental decisions resulting in the state. Similar to Zhang et al. (2016) and Cai and Zhao (2016), our model is a global structural model, using the overall score to disambiguate states, which correspond to sequences of inter-dependent transition actions.
Different from previous work, the structure of our scoring network is shown in Figure 1. It consists of three main layers. On the bottom is a representation layer, which derives dense representations XW , XP and XC for W,P and C, respectively. We compare various distributed representations and neural network structures for learning XW , XP and XC , detailed in Section 3.1. On top of the representation layer, we use a hidden layer to merge XW , XP and XC into a single vector
h = tanh(WhW ·XW +WhP ·XP +WhC ·XC+bh) (1) The hidden feature vector h is used to represent the state S = 〈W,P,C〉, for calculating the scores of the next action. In particular, a linear output layer with two nodes is employed:
o = Wo · h+ bo (2)
The first and second node of o represent the scores of SEP and APP given S, namely Score(S, SEP), Score(S,APP) respectively.
3.1 Representation Learning
Characters. We investigate two different approaches to encoding incoming characters, namely
1An end of sentence symbol 〈/s〉 is added to the input so that the last partial word can be put onto W as a full word before segmentation finishes.
a window approach and an LSTM approach. For the former, we follow prior methods (Xue et al., 2003; Pei et al., 2014), using five-character window [c−2, c−1, c0, c1, c2] to represent incoming characters. Shown in Figure 3, multi-layer perceptron (MLP) is employed to derive a five-character window vector DC from single-character vector representations Vc−2 , Vc−1 , Vc0 , Vc1 , Vc2 .
DC = MLP([Vc−2 ;Vc−1 ;Vc0 ;Vc1 ;Vc2 ]) (3)
For the latter, we follow recent work (Chen et al., 2015b; Zhang et al., 2016), using a bidirectional LSTM to encode input character sequence.2 In particular, the bi-directional LSTM hidden vector [ ←− hC(c0); −→ hC(c0)] of the next incoming character c0 is used to represent the coming characters [c0, c1, ...] given a state. Intuitively, a five-character window provides a local context from which the meaning of the middle character can be better disambiguated. LSTM, on the other hand, captures larger contexts, which can contain more useful clues for dismbiguation but also irrelevant information. It is therefore interesting to investigate a combination of their strengths, by first deriving a locally-disambiguated version of c0, and then feed it to LSTM for a globally disambiguated representation.
Now with regard to the single-character vector representation Vci(i ∈ [−2, 2]), we follow previous work and consider both character embedding ec(ci) and character-bigram embedding eb(ci, ci+1) , investigating the effect of each on the accuracies. When both ec(ci) and eb(ci, ci+1) are utilized, the concatenated vector is taken as Vci . Partial Word. We take a very simple approach to representing the partial word P , by using the embedding vectors of its first and last characters, as well as the embedding of its length. Length embeddings are randomly initialized and then tuned in model training. XP has relatively less influence on the empirical segmentation accuracies.
XP = [e c(P [0]); ec(P [−1]); el(LEN(P ))] (4)
Word. Similar to the character case, we investigate two different approaches to encoding incoming characters, namely a window approach and an LSTM approach. For the former, we follow prior
2The LSTM variation with coupled input and forget gate but without peephole connections is applied (Gers and Schmidhuber, 2000)
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
. . .
. . . . . . . . . .  .  . . . .
MLP
... ...... punc. silver hete. POS
shared parameters
main training
pretrainingBi-LSTM
S A
hidden layer
output
... ... ...
...
XW XP XC
h
DC
Vc-2 Vc-1 Vc0 Vc1 Vc2
Figure 3: Shared character representation.
methods (Zhang and Clark, 2007; Sun, 2010), using the two-word window [w−2, w−1] to represent recognized words. A hidden layer is employed to derive a two-word vector XW from single word embeddings ew(w−2) and ew(w−1).
XW = tanh(Ww[e w(w−2); e w(w−1)] + bw) (5)
For the latter, we follow Zhang et al. (2016) and Cai and Zhao (2016), using an uni-directional LSTM on words that have been recognized.
3.2 Pretraining
Neural network models for NLP benefit from pretraining of word/character embeddings, learning distributed sementic information from large raw texts for reducing sparsity. The three basic elements in our neural segmentor, namely characters, character bigrams and words, can all be pretrained over large unsegmented data. We pretrain the fivecharacter window network in Figure 3 as an unit, learning the MLP parameter together with character and bigram embeddings. We consider four types of commonly explored external data to this end, all of which have been studied for statistical word segmentation, but not for neural network segmentors. Raw Text. Although raw texts do not contain explicit word boundary information, statistics such as mutual information between consecutive characters can be useful features for guiding segmentation (Sun and Xu, 2011). For neural segmentation, these distributional statistics can be implicitly learned by pretraining character embeddings. We therefore consider a more explicit clue for pretraining our character window network, namely punctuations (Li and Sun, 2009).
Punctuation can serve as a type of explicit markup (Spitkovsky et al., 2010), indicating that the two characters on its left and right belong to two
different words. We leverage this source of information by extracting character five-grams excluding punctuation from raw sentences, using them as inputs to classify whether there is punctuation before middle character. Denoting the resulting five character window as [c−2, c−1, c0, c1, c2], the MLP in Figure 3 is used to derive its representation DC , which is then fed to a softmax layer for binary classification:
P (punc) = softmax(Wpunc ·DC + bpunc) (6)
Here P (punc) indicates the probability of a punctuation mark existing before c0. Standard backpropagation training of the MLP in Figure 3 can be done jointly with the training of Wpunc and bpunc. After such training, the embedding Vci and MLP values can be used to initialize the corresponding parameters for DC in the main segmentor, before its training. Automatically Segmented Text. Large texts automatically segmented by a baseline segmentor can be used for self-training (Liu and Zhang, 2012) or deriving statistical features (Wang et al., 2011). We adopt a simple strategy, taking automatically segmented text as silver data to pretrain the five-character window network. Given [c−2, c−1, c0, c1.c2], DC is derived using the MLP in Figure 3, and then used to classify the segmentation of c0 into B(begining)/M(middle)/E(end)/S(single character word) labels.
P (silver) = softmax(Wsilv ·DC + bsilv) (7)
Here Wsilv and bsilv are model parameters. Training can be done in the same way as training with punctuation. Heterogenous Training Data. Multiple segmentation corpora exist for Chinese, with different segmentation granularities. There has been investigation on leveraging two corpora under different annotation standards to improve statistical segmentation (Jiang et al., 2009). We try to utilize heterogenous treebanks by taking an external treebank as labeled data, training a B/M/E/S classifier for the character windows network.
P (hete) = softmax(Whete ·DC + bhete) (8)
POS Data. Previous research has shown that POS information is closely related to segmentation (Ng
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Paramater Value Paramater Value α 0.01 size(ec) 50 λ 10−8 size(eb) 50 p 0.2 size(ew) 50 η 0.2 size(el) 20 MLP layer 2 size(XC) 150 beam B 8 size(XP ) 50 size(h) 200 size(XW ) 100
Table 1: Hyper-parameter values.
and Low, 2004; Zhang and Clark, 2008). We verify the utility of POS information for our segmentor by pretraining a classifier that predicts the POS on each character, according to the character window representation DC . In particular, given [c−2, c−1, c0, c1, c2], the POS of the word that c0 belongs to is used as the output.
P (pos) = softmax(Wpos ·DC + bpos) (9)
Multitask Learning. While each type of external training data can offer one source of segmentation information, different external data can be complimentary to each other. We aim to inject all sources of information into the character window representation DC by using it as a shared representation for different classification tasks. Neural model have been shown capable of doing multi-task learning via parameter sharing (Collobert et al., 2011). Shown in Figure 3, in our case, the output layer for each task is independent, but the hidden layer DC and all layers below DC are shared. We randomly sample from different training sources according to their sizes, performing mixed training.
4 Decoding and Training
To train the main segmentor, we adopt the global transition-based learning and beam-search strategy of Zhang and Clark (2011). For decoding, standard beam search is used, where the B best partial output hypotheses at each step are maintained in an agenda. Initially, the agenda contains only the start state. At each step, all hypotheses in the agenda are expanded, by applying all possible actions and B highest scored resulting hypotheses are used as the agenda for the next step.
For training, the same decoding process is applied to each training example (xi, yi). At step j, if the gold-standard sequence of transition actions yij falls out of the agenda, max-margin update is
Algorithm 1: Training Input : (xi, yi) Parameters: Θ Process: agenda← (S = 〈[ ], φ,Xi〉, V = 0) for j in [0:LEN(Xi)] do
beam = [] for ŷ in agenda do
ŷ′ = ACTION(ŷ, SEP) ADD(ŷ′, beam) ŷ′ = ACTION(ŷ, APP) ADD(ŷ′, beam) end agenda← TOP(beam, B) if yij /∈ agenda then
ŷj = BESTIN(agenda) UPDATE(yij , ŷj ,Θ) return
end end ŷ = BESTIN(agenda) UPDATE(yi, ŷ,Θ) return
performed by taking the current best hypothesis ŷj in the beam as a negative example, and yij as a positive example. The loss function is
l(ŷj , y i j) = (score(ŷj) + η · δ(ŷj , yij))
− score(yij), (10)
where δ(ŷj , yij) is the number of incorrect local decisions in ŷj , and η controls the score margin.
The strategy above is early-update (Collins and Roark, 2004). On the other hand, if the goldstandard hypothesis does not fall out of the agenda until the full sentence has been segmented, yet does not score the highest in the agenda after the last step, a final update is made between the highest scored hypothesis ŷ in the agenda and the goldstandard yi, using exactly the same loss function. Pseudocode for the online learning algorithm is shown in Algorithm 1.
We use Adagrad (Duchi et al., 2011) to optimize model parameters, with an initial learning rate α. L2 regularization and dropout (Srivastava et al., 2014) on input are used to reduce overfitting, with a L2 weight λ and a dropout rate p. All the parameters in our model are randomly initialized to a value (−r, r), where r = √ 6.0
fanin+fanout (Ben-
gio, 2012). We fine-tune character and character
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Source #Chars #Words #Sents Raw data Gigaword 116.5m – – Auto seg Gigaword 398.2m 238.6m 12.04m
Hete. People’s Daily 10.14m 6.17m 104k POS People’s Daily 10.14m 6.17m 104k
Table 2: Statistics of external data.
bigram embeddings, but not word embeddings, acccording to Zhang et al. (2016).
5 Experiments
5.1 Experimental Settings
Data. We use Chinese Treebank 6.0 (CTB6) (Xue et al., 2005) as our main dataset. Training, development and test set splits follow previous work (Zhang et al., 2014). In order to verify the robustness of our model, we additionally use SIGHAN 2005 bake-off (Emerson, 2005) and NLPCC 2016 shared task for Weibo segmentation (Qiu et al., 2016) as test datasets, where the standard splits are used. For pretraining embedding of words, characters and character bigrams, we use Chinese Gigaword (simplified Chinese sections)3, automatically segmented using ZPar 0.6 off-theshelf (Zhang and Clark, 2007), the statictics of which are shown in Table 2.
For pretraining character representations, we extract punctuation classification data from the Gigaword corpus, and use the word-based ZPar and a standard character-based CRF model (Tseng et al., 2005) to obtain automatic segmentation results. We compare pretraining using ZPar results only and using results that both segmentors agree on. For heterogenous segmentation corpus and POS data, we use a People’s Daily corpus of 5 months4. Statistics are listed in Table 2. Evaluation. The standard word precision, recall and F1 measure (Emerson, 2005) are used to evaluate segmentation performances. Hyper-parameter Values. We adopt commonly used values for most hyperparameters, but tuned the sizes of hidden layers on the development set. The values are summarized in Table 1.
5.2 Development Experiments
We perform development experiments to verify the usefulness of various context representations, network configurations and different pretraining methods, respectively.
3https://catalog.ldc.upenn.edu/LDC2011T13 4http://www.icl.pku.edu.cn/icl res
5.2.1 Context Representations
The influence of character and word context representations are empirically studied by varying the network structures forXC andXW in Figure 1, respectively. All the experiments in this section are performed using a beam size of 8. Character Context. We fix the word representation XW to a 2-word window and compare different character context representations. The results are shown in Table 3, where “no char” represents our model without XC , “5-char window” represents a five-character window context, “char LSTM” represents character LSTM context and “5-char window + LSTM” represents a combination, detailed in Section 3.1. “-char emb” and “- bichar emb” represent the combined window and LSTM context without character and characterbigram information, respectively.
As can be seen from the table, without character information, the F-score is 84.62%, demonstrating the necessity of character contexts. Using window and LSTM representations, the Fscores increase to 95.41% and 95.51%, respectively. A combination of the two lead to further improvement, showing that local and global character contexts are indeed complementary, as hypothesized in Section 3.1. Finally, by removing character and character-bigram embeddings, the F-score decreases to 95.20% and 94.27%, respectively, which suggests that character bigrams are more useful compared to character unigrams. This is likely because they contain more distinct tokens and hence offer a larger parameter space. Word Context. The influence of various word contexts are shown in Table 4. Without using word information, our segmentor gives an F-score of 95.66% on the development data. Using a context of only w−1 (1-word window), the F-measure increases to 95.78%. This shows that word contexts are far less important in our model compared to character contexts, and also compared to word contexts in previous word-based segmentors
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Word P R F No word 95.50 95.83 95.66 1-word window 95.70 95.85 95.78 2-word window 95.77 95.95 95.86 3-word window 95.80 95.85 95.83
word LSTM 95.71 95.97 95.84 2-word window+LSTM 95.74 95.95 95.84
Table 4: Influence of word contexts.
(Zhang et al., 2016; Cai and Zhao, 2016). This is likely due to the difference in our neural network structures, and that we fine-tune both character and character bigram embeddings, which significantly enlarges the adjustable parameter space as compared with Zhang et al. (2016). The fact that word contexts can contribute relatively less than characters in a word is also not surprising in the sense that word-based neural segmentors do not outperform the best character-based models by large margins. Given that character context is what we pretrain, our model relies more heavily on them.
With both w−2 and w−1 being used for the context, the F-score further increases to 95.86%, showing that a 2-word window is useful by offering more contextual information. On the other hand, when w−3 is also considered, the F-score does not improve further. This is consistent with previous findings of statistical word segmentation (Zhang and Clark, 2007), which adopt a 2-word context. Interestingly, using a word LSTM does not bring further improvements, even when it is combined with a window context. This suggests that global word contexts may not offer crucial additional information compared with local word contexts. Intuitively, words are significantly less polysemous compared with characters, and hence can serve as effective contexts even if used locally, to supplement a more crucial character context.
5.2.2 Stuctured Learning and Inference
We verify the effectiveness of structured learning and inference by measuring the influence of beam size on the baseline segmentor. Figure 4 shows the F-scores against different numbers of training iterations with beam size 1,2,4,8 and 16, respectively. When the beam size is 1, the inference is local and greedy. As the size of the beam increases, more global structural ambiguities can be resolved since learning is designed to guide search. A contrast between beam sizes 1 and 2 demonstrates the usefulness of structured learning and inference. As
Pretrain P R F ER% Baseline 95.77 95.95 95.86 0
+Punc. pretrain 96.36 96.13 96.25 -9.4 +Auto-seg pretrain 96.23 96.29 96.26 -9.7 +Heter-seg pretrain 96.28 96.27 96.27 -9.9
+POS pretrain 96.16 96.28 96.22 -8.7 +Multitask pretrain 96.49 96.39 96.44 -14.0
Table 5: Influence of pretraining.
the beam size increases, the gain by doubling the beam size decreases. We choose a beam size of 8 for the remaining experiments for a tradeoff between speeds and accuraces.
5.2.3 Pretraining Results
Table 5 shows the effectiveness of rich pretraining of Dc on the development set. In particular, by using punctuation information, the F-score increases from 95.86% to 96.25%, with a relative error reduction of 9.4%. This is consistent with the observation of Sun and Xu (2011), who show that punctuation is more effective compared with mutual information and access variety as semisupervised data for a statistical word segmentation model. With automatically-segmented data5, heterogenous segmentation and POS information, the F-score increases to 96.26%, 96.27% and 96.22%, respectively, showing the relevance of all information sources to neural segmentation, which is consistent with observations made for statistical word segmentation (Jiang et al., 2009; Wang et al., 2011; Zhang et al., 2013). Finally, by integrating all above information via multi-task learning, the F-score is further improved to 96.44%, with a 14% relative error reduction.
5By using ZPar alone, the auto-segmented result is 96.02%, less than using results by matching ZPar and the CRF segmentor outputs.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Models P R F Baseline 95.3 95.5 95.4 Multitask pretrain 96.2 96.0 96.1 Sun and Xu (2011) baseline 95.2 94.9 95.1 Sun and Xu (2011) multi-source semi 95.9 95.6 95.7 Zhang et al. (2016) neural 95.3 94.7 95.0
Zhang et al. (2016)* hybrid 96.1 95.8 96.0 Chen et al. (2015a) window 95.7 95.8 95.8 Chen et al. (2015b) char LSTM 96.2 95.8 96.0 Zhang et al. (2014) POS and syntax – – 95.7 Wang et al. (2011) statistical semi 95.8 95.8 95.8 Zhang and Clark (2011) statistical 95.5 94.8 95.1
Table 6: Main results on CTB6.
5.3 Final Results
Our final results on CTB6 are shown in Table 6, which lists the results of several current state-ofthe-art methods. Without multitask pretraining, our model gives an F-score of 95.44%, which is higher than the neural segmentor of Zhang et al. (2016), which gives the best accuracies among pure neural segments on this dataset. By using multitask pretraining, the result increases to 96.09%, with a relative error reduction of 14.5%. In comparison, Sun and Xu (2011) investigated heterogenous semi-supervised learning on a stateof-the-art statistical model, obtaining a relative error reduction of 13.8%. Our findings show that external data can be as useful for neural segmentation as for statistical segmentation.
Our final results compares favourably to the best statistical models, including those using semisupervised learning (Sun and Xu, 2011; Wang et al., 2011), and those leveraging joint POS and syntactic information (Zhang et al., 2014). In addition, it also outperforms the best neural models, in particular Zhang et al. (2016)*, which is a hybrid neural and statistical model, integrating manual discrete features into their word-based neural model. We achieve the best reported F-score on this dataset. To our knowledge, this is the first time a pure neural network model outperforms all existing methods on this dataset, allowing the use of external data.6
In addition to CTB6, which has been the most commonly adopted by recent segmentation research, we additionally evaluate our results on the
6 We did not investigate the use of lexicons (Chen et al., 2015a,b) in our research, since lexicons might cover different OOV in the training and test data, and hence directly affecting the accuracies, which makes it relatively difficult to compare different methods fairly unless a single lexicon is used for all methods, as observed by Cai and Zhao (2016).
F1 measure PKU MSR AS CityU Weibo Multitask pretrain 96.2 97.3 95.6 96.7 95.4
Cai and Zhao (2016) 95.5 96.5 – – – Zhang et al. (2016) 95.1 97.0 – – – Zhang et al. (2016)* 95.7 97.7 – – – Pei et al. (2014) 95.2 97.2 – – – Sun et al. (2012) 95.4 97.4 – – – Zhang and Clark (2007) 94.5 97.2 94.6 95.1 – Zhang et al. (2006) 95.1 97.1 95.1 95.1 – Sun et al. (2009) 95.2 97.3 – 94.6 –
Sun (2010) 95.2 96.9 95.2 95.6 – Wang et al. (2014) 95.3 97.4 95.4 94.7 – Xia et al. (2016) – – – – 95.4
Table 7: Main results on other test datasets.
SIGHAN 2005 bakeoff and Weibo datasets, to examine cross domain robustness. Different stateof-the-art methods for which results are recorded on these datasets are listed in Table 7. Most neural models reported results only on the PKU and MSR datasets of the bakeoff test sets, which are in simplified Chinese. The AS and CityU corpora are in traditional Chinese, sourced from Taiwan and Hong Kong corpora, respectively. We map them into simplified Chinese before segmentation. The Weibo corpus is in a yet different genre, being social media text. Xia et al. (2016) achieved the best results on this dataset by using a statistical model with features learned using external lexicons, the CTB7 corpus and the People Daily corpus. Similar to Table 6, our method gives the best accuracies on all corpora except for MSR, where it underperforms the hybrid model of Zhang et al. (2016) by 0.4%. To our knowledge, we are the first to report results for a neural segmentor on more than 3 datasets, with competitive results consistently. It verifies that knowledge learned from a certain set of resources can be used to enhance cross-domain robustness in training a neural segmentor for different datasets, which is of practical importance.
6 Conclusion
We investigated rich external resources for enhancing neural word segmentation, by building a globally optimised beam-search model that leverages both character and word contexts. Taking each type of external resource as an auxiliary classification task, we use neural multi-task learning to pre-train a set of shared parameters for character contexts. Results show that rich pretraining leads to 14.5% relative error reduction, and our model gives results highly competitive to the best systems on six different benchmarks.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths: i. Well organized and easy to understand ii. Provides detailed comparisons under various experimental settings and shows the state-of-the-art performances  - Weaknesses: i. In experiments, this paper compares previous supervised approaches, but the proposed method is the semi-supervised approach even if the training data is enough to train.  - General Discussion: This paper adopts a pre-training approach to improve Chinese word segmentation. Based on the transition-based neural word segmentation, this paper aims to pre-train incoming characters with external resources (punctuation, soft segmentation, POS, and heterogeneous training data) through multi-task learning. That is, this paper casts each external source as an auxiliary classification task. The experimental results show that the proposed method achieves the state-of-the-art performances in six out of seven datasets. u00a0  This paper is well-written and easy to understand. A number of experiments prove the effectiveness of the proposed method. However, there exist an issue in this paper. The proposed method is a semi-supervised learning that uses external resources to pre-train the characters. Furthermore, this paper uses another heterogeneous training datasets even if it uses the datasets only for pre-training. Nevertheless, the baselines in the experiments are based on supervised learning. In general, the performance of semi-supervised learning is better than that of supervised learning because semi-supervised learning makes use of plentiful auxiliary information. In the experiments, this paper should have compared the proposed method with semi-supervised approaches.  POST AUTHOR RESPONSE  What the reviewer concerned is that this paper used additional  gold-labeled d dataset to pretrain the character embeddings. Some baselines in the experiments used label information, where the labels are predicted automatically by their base models as the authors pointed out. When insisting superiority of a method, all circumstances should be same. Thus, even if the gold dataset isn t used to train the segmentation model directly, it seems to me that it is an unfair comparison because the proposed method used another  gold d dataset to train the character embeddings.