Different linguistic perspectives causes many diverse segmentation criteria for Chinese word segmentation (CWS). Most existingmethods focus on improve the performance for each single criterion. However, it is interesting to exploit these different criteria and mining their common underlying knowledge. In this paper, we propose adversarial multi-criteria learning for CWS by integrating shared knowledge frommultiple heterogeneous segmentation criteria. Experiments on eight corpora with heterogeneous segmentation criteria show that the performance of each corpus obtains a significant improvement, compared to single-criterion learning.
1 Introduction
Chinese word segmentation (CWS) is a preliminary and important task for Chinese natural language processing (NLP). Currently, the state-ofthe-art methods are based on statistical supervised learning algorithms (Xue, 2003; Zhao et al., 2006), and rely on a large-scale annotated corpus whose cost is extremely expensive. Although there have been great achievements in building CWS corpora, they are somewhat incompatible due to different segmentation criteria. As shown in Table 1, given a sentence “姚明进入总决赛 (YaoMing reaches the final)”, the two commonly-used corpora, PKU’s People’s Daily (PKU) (Yu et al., 2001) and Penn Chinese Treebank (CTB) (Xia, 2000), use different segmentation criteria. In a sense, it is a waste of resources if we fail to fully exploit these corpora. Recently, some efforts have been made to exploit heterogeneous annotation data for Chinese word segmentation or part-of-speech tagging
Corpora Yao Ming reaches the final CTB 姚明 进入 总决赛 PKU 姚 明 进入 总 决赛
(Jiang et al., 2009; Sun and Wan, 2012; Qiu et al., 2013; Li et al., 2015). These methods adopted stacking or multi-task architectures and showed that heterogeneous corpora can help each other. However, most of these model adopt the shallow linear classifier with discrete features, which makes it difficult to design the shared feature spaces, usually resulting in a complex model. Fortunately, recent deep neural models provide a convenient way to share information among multiple tasks (Collobert and Weston, 2008; Luong et al., 2015; Chen et al., 2016). In this paper, we propose an adversarial multicriteria learning for CWS by integrating shared knowledge from multiple segmentation criteria. Specifically, we regard each segmentation criterion as a single task and propose three different shared-private models under the framework of multi-task learning (Caruana, 1997; Ben-David and Schuller, 2003), where a shared layer is used to extract the criteria-invariant features, and a private layer is used to extract the criteria-specific features. Inspired by the success of adversarial strategy on domain adaption (Ajakan et al., 2014; Ganin et al., 2016; Bousmalis et al., 2016), we further utilize adversarial strategy to make sure the shared layer can extract the common underlying and criteria-invariant features, which are suitable for all the criteria. Finally, we exploit the eight segmentation criteria on the five simplified Chinese and three traditional Chinese corpora. Experiments show that our models are effective to improve the performance for CWS. We also observe
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
that traditional Chinese could benefit from incorporating knowledge from simplified Chinese. The contributions of this paper could be summarized as follows.
• Multi-criteria learning is first introduced for CWS, in which we propose three sharedprivate models to integrate multiple segmentation criteria. • An adversarial strategy is used to force the shared layer to learn criteria-invariant features, in which an new objective function is also proposed instead of the original crossentropy loss. • We conduct extensive experiments on eight CWS corpora with different segmentation criteria, which is by far the largest number of datasets used simultaneously.
2 General Neural Model for Chinese Word Segmentation
Chinese word segmentation task is usually regarded as a character based sequence labeling problem. Specifically, each character in a sentence is labeled as one of L = {B,M,E, S}, indicating the begin, middle, end of a word, or a word with single character. There are lots of prevalent methods to solve sequence labeling problem such as maximum entropy Markov model (MEMM), conditional random fields (CRF), etc. Recently, neural networks are widely applied to Chinese word segmentation task for their ability to minimize the effort in feature engineering (Zheng et al., 2013; Pei et al., 2014; Chen et al., 2015; Ma and Hinrichs, 2015; Xu and Sun, 2016; Yao and Huang, 2016; Cai and Zhao, 2016; Zhang et al., 2016). Specifically, given a sequence with n characters X = {x1, . . . , xn}, the aim of CWS task is to figure out the ground truth of labels Y ∗ = {y∗1, . . . , y∗n}:
Y ∗ = argmax Y ∈Ln p(Y |X), (1)
where L = {B,M,E, S}. The general architecture of neural CWS could be characterized by three components: (1) a character embedding layer; (2) feature layers consisting of several classical neural networks and (3) a tag inference layer. The role of feature layers is to extract features, which could be either convolution neural network or recurrent neural network. In this paper, we employ the state-of-the-art architecture
(Huang et al., 2015; Ma and Hovy, 2016) which adopts the bi-direction long short-term memory with CRF as tag inference layer. Figure 1 illustrates the general architecture of CWS.
2.1 Embedding layer
In neural models, the first step usually is to map discrete language symbols to distributed embedding vectors. Formally, we lookup embedding vector from embedding matrix for each character xi as exi ∈ Rde , where de is a hyper-parameter indicating the size of character embedding.
2.2 Feature layers
We adopt bi-directional long short-term memory (Bi-LSTM) as feature layers. While there are numerous LSTM variants, here we use the LSTM architecture used by (Jozefowicz et al., 2015), which is similar to the architecture of (Graves, 2013) but without peep-hole connections.
LSTM LSTM introduces gate mechanism and memory cell to maintain long dependency information and avoid gradient vanishing. Formally, LSTM, with input gate i, output gate o, forget gate f and memory cell c, could be expressed as:
ii oi fi c̃i
 =
σ σ σ ϕ
(Wgᵀ [ exihi−1 ] + bg ) , (2)
ci = ci−1 ⊙ fi + c̃i ⊙ ii, (3) hi = oi ⊙ ϕ(ci), (4)
where Wg ∈ R(de+dh)×4dh and bg ∈ R4dh are trainable parameters. dh is a hyper-parameter, in-
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
dicating the hidden state size. Function σ(·) and ϕ(·) are sigmoid and tanh functions respectively.
Bi-LSTM In order to incorporate information from both sides of sequence, we use bi-directional LSTM (Bi-LSTM) with forward and backward directions. The update of each Bi-LSTM unit can be written precisely as follows:
hi = −→h i ⊕ ←−h i, (5)
= Bi-LSTM(exi , −→h i−1, ←−h i+1, θ), (6)
where −→h i and ←−h i are the hidden states at position i of the forward and backward LSTMs respectively; ⊕ is concatenation operation; θ denotes all parameters in Bi-LSTM model.
2.3 Inference Layer
After extracting features, we employ conditional random fields (CRF) (Lafferty et al., 2001) layer to inference tags. In CRF layer, p(Y |X) in Eq (1) could be formalized as:
p(Y |X) = Ψ(Y |X)∑ Y ′∈Ln Ψ(Y ′|X) . (7)
Here, Ψ(Y |X) is the potential function, and we only consider interactions between two successive labels (first order linear chain CRFs):
Ψ(Y |X) = n∏
i=2
ψ(X, i, yi−1, yi), (8)
ψ(x, i, y′, y) = exp(s(X, i)y + by′y), (9)
where by′y ∈ R is trainable parameters respective to label pair (y′, y). Score function s(X, i) ∈ R|L| assigns score for each label on tagging the i-th character:
s(X, i) = W⊤s hi + bs, (10)
where hi is the hidden state of Bi-LSTM at position i; Ws ∈ Rdh×|L| and bs ∈ R|L| are trainable parameters.
3 Multi-Criteria Learning for Chinese Word Segmentation
Although neural models are widely used on CWS, most of them cannot deal with incompatible criteria with heterogonous segmentation criteria simultaneously. Inspired by the success of multi-task learning (Caruana, 1997; Ben-David and Schuller, 2003),
we regard the heterogenous criteria asmultiple “related” tasks, which could improve the performance of each other simultaneously with shared information. Formally, assume that there areM corpora with heterogeneous segmentation criteria. We referDm as corpusm with Nm samples:
Dm = {(X(m)i , Y (m) i )} Nm i=1, (11)
where Xmi and Y mi denote the i-th sentence and the corresponding label in corpusm. To exploit the shared information between these different criteria, we propose three sharing models for CWS task as shown in Figure 2. The feature layers of these three models consist of a private (criterion-specific) layer and a shared (criterioninvariant) layer. The difference between three models is the information flow between the task layer and the shared layer. Besides, all of these three models also share the embedding layer.
3.1 Model-I: Parallel Shared-Private Model
In the feature layer of Model-I, we regard the private layer and shared layer as two parallel layers. For corpusm, the hidden states of shared layer and private layer are:
h(s)i =Bi-LSTM(exi , −→h (s)i−1, ←−h (s)i+1, θs), (12)
h(m)i =Bi-LSTM(exi , −→h (m)i−1, ←−h (m)i+1, θm), (13)
and the score function in the CRF layer is computed as:
s(m)(X, i) = W(m)s ⊤ [ h(s)i h(m)i ] + b(m)s , (14)
where W(m)s ∈ R2dh×|L| and b(m)s ∈ R|L| are criterion-specific parameters for corpusm.
3.2 Model-II: Stacked Shared-Private Model
In the feature layer of Model-II, we arrange the shared layer and private layer in stacked manner. The private layer takes output of shared layer as input. For corpus m, the hidden states of shared layer and private layer are:
h(s)i = Bi-LSTM(exi , −→h (s)i−1, ←−h (s)i+1, θs), (15)
h(m)i = Bi-LSTM( [ exi h(s)i ] , −→h (m)i−1, ←−h (m)i+1 , θm) (16)
and the score function in the CRF layer is computed as:
s(X, i) = W(m)s ⊤ h(m)i + b (m) s , (17)
where W(m)s ∈ R2dh×|L| and b(m)s ∈ R|L| are criterion-specific parameters for corpusm.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
CRF
CRF
Task A
Task B
X(A)
X(B)
Y(B)
Y(A)
(a) Model-I
CRF
CRF
Task A
Task B
X(A)
X(B)
Y(B)
Y(A)
(b) Model-II
CRF
CRF
Task A
Task B
X(A)
X(B)
Y(B)
Y(A)
(c) Model-III
Figure 2: Three shared-private models for multi-criteria learning. The yellow blocks are the shared BiLSTM layer, while the gray block are the private Bi-LSTM layer. The yellow circles denote the shared embedding layer. The red information flow indicates the difference between three models.
3.3 Model-III: Skip-Layer Shared-Private Model
In the feature layer of Model-III, the shared layer and private layer are in stacked manner as ModelII. Additionally, we send the outputs of shared layer to CRF layer directly. The Model III can be regarded as a combination ofModel-I andModel-II. For corpusm, the hidden states of shared layer and private layer are the same with Eq (15) and (16), and the score function in CRF layer is computed as the same as Eq (14).
3.4 Objective function
The parameters of the network are trained to maximize the log conditional likelihood of true labels on all the corpora. The objective functionJseg can be computed as:
Jseg(Θm,Θs) = M∑
m=1 Nm∑ i=1 log p(Y (m)i |X (m) i ; Θ m,Θs),
(18) whereΘm andΘs denote all the parameters in private and shared layers respectively.
4 Incorporating Adversarial Training for Shared Layer
Although the shared-private model separates the feature space into shared and private spaces, there is no guarantee that sharable features do not exist in private feature space, or vice versa. Inspired by the work on domain adaptation (Ajakan et al., 2014; Ganin et al., 2016; Bousmalis et al., 2016), we hope that the features extracted by shared layer is invariant across the heterogonous segmentation criteria. Therefore, we jointly optimize the shared
layer via adversarial training (Goodfellow et al., 2014). Therefore, besides the task loss for CWS, we additionally introduce an adversarial loss to prevent criterion-specific feature from creeping into shared space as shown in Figure 3. We use a criterion discriminator which aims to recognizewhich criterion the sentence is annotated by using the shared features. Specifically, given a sentence X with length n, we refer to h(s)X as shared features for X in one of the sharing models. Here, we compute h(s)X by simply averaging the hidden states of shared layer h(s)X = 1 n ∑n i h (s) xi . The criterion discriminator computes the probability p(·|X) over all criteria as:
p(·|X; Θd,Θs) = softmax(W⊤d h (s) X + bd), (19)
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
whereΘd indicates the parameters of criterion discriminator Wd ∈ Rdh×M and bd ∈ RM ; Θs denotes the parameters of shared layers.
4.1 Adversarial loss function
The criterion discriminator maximizes the cross entropy of predicted criterion distribution p(·|X) and true criterion.
max Θd J 1adv(Θd) = M∑ m=1 Nm∑ i=1 log p(m|X(m)i ; Θ d,Θs). (20)
An adversarial loss aims to produce shared features, such that a criterion discriminator cannot reliably predict the criterion by using these shared features. Therefore, we maximize the entropy of predicted criterion distribution when training shared parameters.
max Θs J 2adv(Θs) = M∑ m=1 Nm∑ i=1 H ( p(m|X(m)i ; Θ d,Θs) ) ,
(21) where H(p) = − ∑ i pi log pi is an entropy of distribution p. Unlike (Ganin et al., 2016), we use entropy term instead of negative cross-entropy.
5 Training
Finally, we combine the task and adversarial objective functions.
J (Θ;D) = Jseg(Θm,Θs) + λJ 1adv(Θd) + λJ 2adv(Θs), (22) where λ is the weight that controls the interaction of the loss terms and D is the training corpora. The training procedure is to optimize two discriminative classifiers alternately as shown in Algorithm 1. We use AdaGrad (Duchi et al., 2011) with minibatchs to maximize the objectives. Notably, when using adversarial strategy, we firstly train 2400 epochs (each epoch only trains on eight batches from different corpora), then we only optimize Jseg(Θm,Θs) with Θs fixed until convergence (early stop strategy).
6 Experiments
6.1 Datasets
To evaluate our proposed architecture, we experiment on eight prevalent CWS datasets from SIGHAN2005 (Emerson, 2005) and SIGHAN2008 (Jin and Chen, 2008). Table 2 gives the details of the eight datasets. Among
Algorithm 1 Adversarial multi-criteria learning for CWS task. 1: for i = 1; i <= n_epoch; i++ do 2: # Train tag predictor for CWS 3: form = 1;m <= M ;m++ do 4: # Randomly pick data from corpusm 5: B = {X,Y }bm1 ∈ Dm 6: Θs += α∇ΘsJ (Θ;B) 7: Θm += α∇ΘmJ (Θ;B) 8: end for 9: # Train criterion discriminator 10: form = 1;m <= M ;m++ do 11: B = {X,Y }bm1 ∈ Dm 12: Θd += α∇ΘdJ (Θ;B) 13: end for 14: end for
Datasets Nw Nc |Dw| |Dc| Ns Si gh an 05 MSRA Train 2.4M 4.1M 88.1K 5.2K 86.9KTest 0.1M 0.2M 12.9K 2.8K 4.0K AS Train 5.4M 8.4M 141.3K 6.1K 709.0KTest 0.1M 0.2M 18.8K 3.7K 14.4K
Si gh an 08
PKU Train 1.1M 1.8M 55.2K 4.7K 47.3KTest 0.2M 0.3M 17.6K 3.4K 6.4K CTB Train 0.6M 1.1M 42.2K 4.2K 23.4KTest 0.1M 0.1M 9.8K 2.6K 2.1K CKIP Train 0.7M 1.1M 48.1K 4.7K 94.2KTest 0.1M 0.1M 15.3K 3.5K 10.9K CITYU Train 1.1M 1.8M 43.6K 4.4K 36.2KTest 0.2M 0.3M 17.8K 3.4K 6.7K
NCC Train 0.5M 0.8M 45.2K 5.0K 18.9KTest 0.1M 0.2M 17.5K 3.6K 3.6K SXU Train 0.5M 0.9M 32.5K 4.2K 17.1KTest 0.1M 0.2M 12.4K 2.8K 3.7K
Table 2: Details of eight datasets. Nw and Nc indicate numbers of tokens and characters respectively. Dw and Dc are the dictionaries of distinguished words and characters respectively. Ns indicates the number of sentences.
these datasets, AS, CITYU and CKIP are traditional Chinese, while the remains, MSRA, PKU, CTB, NCC and SXU, are simplified Chinese. We use 10% data of shuffled train set as development set for all datasets.
6.2 Experimental Configurations
Table 3 gives the details of the hyper-parameter configurations. Since the scale of each datasets varies, we use different training batch sizes for datasets. Specifically, we set batch sizes of AS and MSR datasets as 512 and 256 respectively, and 128 for remains. We employ dropout strategy on embedding layer, keeping 80% inputs (20% dropout rate).
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Character embedding size de = 50 Initial learning rate α = 0.2
Loss weight coefficient λ = 0.05 LSTM dimensionality dh = 100 Dropout rate on input layer p = 20% Batch size 20
Table 3: Configurations of Hyper-parameters.
For initialization, we random all parameters following uniform distribution at (−0.05, 0.05). We simply map traditional Chinese characters to simplified Chinese, and optimize on the same character embedding matrix across datasets, which is pre-trained on Chinese Wikipedia corpus, using word2vec toolkit (Mikolov et al., 2013). Following previous works (Chen et al., 2015; Pei et al., 2014), all experiments including baseline results are using pre-tarined character embedding with bigram feature.
6.3 Overall Results
Table 4 shows the experiment results of the proposed models on test sets of eight CWS datasets, which has three blocks. (1) In the first block, we can see that the performance of Bi-LSTM cannot be improved by merely increasing the depth of networks. (2) In the second block, our proposed threemodels based on multi-criteria learning boost performance. Model-I gains 0.75% improvement on averaging F-measure score compared with Bi-LSTM result (94.14%). Only the performance on MSRA drops slightly. Compared to the baseline results (Bi-LSTM and stacked Bi-LSTM), the proposed models boost the performance with the help of exploiting information across these heterogeneous segmentation criteria. Although various criteria have different segmentation granularities, there are still some underlying information shared. For instance, MSRA and CTB treat family name and last name as one token “宁泽涛 (NingZeTao)”, whereas some other datasets, like PKU, regard them as two tokens, “宁 (Ning)” and “泽涛 (ZeTao)”. The partial boundaries (before “宁 (Ning)” or after “涛 (Tao)”) can be shared. (3) In the third block, we introduce adversarial training. By introducing adversarial training, the performances are further boosted, and Model-I is slightly better than Model-II and Model-III. The adversarial training tries to make shared layer keep
criteria-invariant features. For instance, as shown in Table 4, when we use shared information, the performance onMSRA drops (worse than baseline result). The reason may be that the shared parameters bias to other segmentation criteria and introduce noisy features into shared parameters. When we additionally incorporate the adversarial strategy, we observe that the performance on MSRA is improved and outperforms the baseline results. We could also observe the improvements on other datasets. However, the boost from the adversarial strategy is not significant. The main reason might be that the proposed three sharing models implicitly attempt to keep invariant features by shared parameters and learn discrepancies by the task layer.
6.4 Traditional & Simplified Chinese
Traditional Chinese and simplified Chinese are two similar languages with slightly difference on character forms and usages on grammar. We investigate that if datasets in traditional Chinese and simplified Chinese could help each other. Table 5 gives the results of Model-I on 3 traditional Chinese datasets under the help of 5 simplified Chinese datasets. Specifically, we firstly train the model on simplified Chinese datasets, then we train traditional Chinese datasets independently with shared parameters fixed. As we can see, the average performance is boosted by 0.41% on F-measure score (from 93.78% to 94.19%), which indicates that shared features learned from simplified Chinese segmentation criteria can help to improve performance on traditional Chinese. Like MSRA, as AS dataset is relatively large (train set of 5.4M tokens), the features learned by shared parameters might bias to other datasets and thus hurt performance on such large dataset AS.
6.5 Speed
To further explore the convergence speed, we plot the results on development sets through epochs. Figure 4 shows the learning curve ofModel-I without incorporating adversarial strategy. As shown in Figure 4, the proposed model makes progress gradually on all datasets. After about 1000 epochs, the performance becomes stable and convergent. We also test the decoding speed, and our models process 441.38 sentences per second averagely. As the proposed models and the baseline models (Bi-LSTM and stacked Bi-LSTM) are nearly in the same complexity, all models are nearly the
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Models MSRA AS PKU CTB CKIP CITYU NCC SXU Avg.
Bi-LSTM
P 95.70 93.64 93.67 95.19 92.44 94.00 91.86 95.11 93.95 R 95.99 94.77 92.93 95.42 93.69 94.15 92.47 95.23 94.33 F 95.84 94.20 93.30 95.30 93.06 94.07 92.17 95.17 94.14 OOV 66.28 70.07 66.09 76.47 72.12 65.79 59.11 71.27 68.40
Stacked Bi-LSTM
P 95.69 93.89 94.10 95.20 92.40 94.13 91.81 94.99 94.03 R 95.81 94.54 92.66 95.40 93.39 93.99 92.62 95.37 94.22 F 95.75 94.22 93.37 95.30 92.89 94.06 92.21 95.18 94.12 OOV 65.55 71.50 67.92 75.44 70.50 66.35 57.39 69.69 68.04 Multi-Criteria Learning
Model-I
P 95.67 94.44 94.93 95.95 93.99 95.10 92.54 96.07 94.84 R 95.82 95.09 93.73 96.00 94.52 95.60 92.69 96.08 94.94 F 95.74 94.76 94.33 95.97 94.26 95.35 92.61 96.07 94.89 OOV 69.89 74.13 72.96 81.12 77.58 80.00 64.14 77.05 74.61
Model-II
P 95.74 94.60 94.82 95.90 93.51 95.30 92.26 96.17 94.79 R 95.74 95.20 93.76 95.94 94.56 95.50 92.84 95.95 94.94 F 95.74 94.90 94.28 95.92 94.03 95.40 92.55 96.06 94.86 OOV 69.67 74.87 72.28 79.94 76.67 81.05 61.51 77.96 74.24
Model-III
P 95.76 93.99 94.95 95.85 93.50 95.56 92.17 96.10 94.74 R 95.89 95.07 93.48 96.11 94.58 95.62 92.96 96.13 94.98 F 95.82 94.53 94.21 95.98 94.04 95.59 92.57 96.12 94.86 OOV 70.72 72.59 73.12 81.21 76.56 82.14 60.83 77.56 74.34 Adversarial Multi-Criteria Learning
Model-I+ADV
P 95.95 94.17 94.86 96.02 93.82 95.39 92.46 96.07 94.84 R 96.14 95.11 93.78 96.33 94.70 95.70 93.19 96.01 95.12 F 96.04 94.64 94.32 96.18 94.26 95.55 92.83 96.04 94.98 OOV 71.60 73.50 72.67 82.48 77.59 81.40 63.31 77.10 74.96
Model-II+ADV
P 96.02 94.52 94.65 96.09 93.80 95.37 92.42 95.85 94.84 R 95.86 94.98 93.61 95.90 94.69 95.63 93.20 96.07 94.99 F 95.94 94.75 94.13 96.00 94.24 95.50 92.81 95.96 94.92 OOV 72.76 75.37 73.13 82.19 77.71 81.05 62.16 76.88 75.16
Model-III+ADV
P 95.92 94.25 94.68 95.86 93.67 95.24 92.47 96.24 94.79 R 95.83 95.11 93.82 96.10 94.48 95.60 92.73 96.04 94.96 F 95.87 94.68 94.25 95.98 94.07 95.42 92.60 96.14 94.88 OOV 70.86 72.89 72.20 81.65 76.13 80.71 63.22 77.88 74.44
Table 4: Results of proposed models on test sets of eight CWS datasets. There are three blocks. The first block consists of two baseline models: Bi-LSTM and stacked Bi-LSTM. The second block consists of our proposed three models without adversarial training. The third block consists of our proposed three models with adversarial training.
AS CKIP CITYU Avg. Base line 94.20 93.06 94.07 93.78 This work 94.12 93.24 95.20 94.19
Table 5: Performance on 3 traditional Chinese datasets with shared parameters fixed shared. The shared parameters are trained on 5 simplified Chinese datasets. Here, we conduct Model-I without incorporating adversarial training strategy.
same efficient. However, the time consumption of training process varies from model to model. For the models without adversarial training, it costs about 10 hours for training (the same for stacked Bi-LSTM to train eight datasets), whereas it takes about 16 hours for the models with adversarial training. All the experiments are conducted on the hardware with Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz and NVIDIA GeForce GTX TITAN
X.
6.6 Error Analysis
We further investigate the benefits of the proposed models by comparing the error distributions between the single-criterion learning (baseline model Bi-LSTM) and multi-criteria learning (Model-I and Model-I with adversarial training) as shown in Figure 5. According to the results, we could observe that a large proportion of points lie above diagonal lines in Figure 5a and Figure 5b, which implies that performance benefit from integrating knowledge and complementary information from other corpora. As shown in Table 4, on the test set of CITYU, the performance of Model-I and its adversarial version (Model-I+ADV) boost from 92.17% to 95.59% and 95.42% respectively. In addition, we observe that adversarial strategy is effective to prevent criterion specific features
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
0 500 1,000 1,500 2,000 2,500
90
92
94
96
epoches
Fva lu e( % )
MSRA AS PKU CTB CKIP CITYU NCC SXU
Figure 4: Convergence speed of Model-I without adversarial training on development sets of eight datasets.
40
50
60
70
80
90
100
40 50 60 70 80 90 100
M u
lt i-
C ri
te ri
a Le
ar n
in g
Base Line (a)
40
50
60
70
80
90
100
40 50 60 70 80 90 100
M u
lt i-
C ri
te ri
a Le
ar n
in g
+ A
d ve
rs ar
y
Base Line (b)
Figure 5: F-measure scores on test set of CITYU dataset. Each point denotes a sentence, with the (x, y) values of each point denoting the F-measure scores of the two models, respectively. (a) is comparison between Bi-LSTM andModel-I. (b) is comparison between Bi-LSTM and Model-I with adversarial training.
from creeping into shared space. For instance, the segmentation granularity of personal name is often different according to heterogenous criteria. With the help of adversarial strategy, our models could correct a large proportion of mistakes on personal name. Table 6 lists the examples from 2333-th and 89-th sentences in test sets of PKU andMSRA datasets respectively.
7 Related Works
There are many works on exploiting heterogeneous annotation data to improve various NLP tasks. Jiang et al. (2009) proposed a stackingbased model which could train a model for one specific desired annotation criterion by utilizing knowledge from corpora with other heterogeneous annotations. Sun and Wan (2012) proposed a
Models PKU-2333 MSRA-89
Golds Lu Wu Xuan Mu Ling Ying 卢 武铉 穆玲英
Base Line 卢武铉 穆 玲英 Model-I 卢武铉 穆 玲英
Modell-I+ADV 卢 武铉 穆玲英
Table 6: Segmentation cases of personal name.
structure-based stacking model to reduce the approximation error, which makes use of structured features such as sub-words. These models are unidirectional aid and also suffer from error propagation problem. Qiu et al. (2013) used multi-tasks learning framework to improve the performance of POS tagging on two heterogeneous datasets. Li et al. (2015) proposed a coupled sequence labeling model which could directly learn and infer two heterogeneous annotations. Chao et al. (2015) also utilize multiple corpora using coupled sequence labeling model. These methods adopt the shallow classifiers, therefore suffering from the problem of defining shared features. Our proposed models use deep neural networks, which can easily share information with hidden shared layers. Chen et al. (2016) also adopted neural network models for exploiting heterogeneous annotations based on neural multi-view model, which can be regarded as a simplified version of our proposed models by removing private hidden layers. Unlike the above models, we design three sharing-private architectures and keep shared layer to extract criterion-invariance features by introducing adversarial training. Moreover, we fully exploit eight corpora with heterogeneous segmentation criteria to model the underlying shared information.
8 Conclusions & Future Works
In this paper, we propose adversarial multi-criteria learning for CWS by fully exploiting the underlying shared knowledge across multiple heterogeneous criteria. Experiments show that our proposed three shared-private models are effective to extract the shared information, and achieve significant improvements over the single-criterion methods.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths:  The authors use established neural network methods (adversarial networks -- Goodfellow et al, NIPS-2014) to take advantage of 8 different Chinese work breaking test sets, with 8 different notions of what counts as a word in Chinese.  This paper could have implications for many NLP tasks where we have slightly different notions of what counts as correct.  We have been thinking of that problem in terms of adaptation, but it is possible that Goodfellow et al is a more useful way of thinking about this problem.  - Weaknesses:  We need a name for the problem mentioned above.  How about: the elusive gold standard.  I prefer that term to multi-criteria.  The motivation seems to be unnecessarily narrow.  The elusive gold standard comes up in all sorts of applications, not just Chinese Word Segmentation.  The motivation makes unnecessary assumptions about how much the reader knows about Chinese.              When you don't know much about something, you think it is easier than it is.  Many non-Chinese readers (like this reviewer) think that Chinese is simpler than it is.              It is easy to assume that Chinese Word Segmentation is about as easy as tokenizing English text into strings delimited by white space.  But my guess is that IAA (inter-annotator agreement) is pretty low in Chinese.  The point you are trying to make in Table 1 is that there is considerable room for disagreement among native speakers of Chinese.  I think it would help if you could point out that there are many NLP tasks where there is considerable room for disagreement.  Some tasks like machine translation, information retrieval and web search have so much room for disagreement that the metrics for those tasks have been designed to allow for multiple correct answers.  For other tasks, like part of speech tagging, we tend to sweep the elusive gold standard problem under a rug, and hope it will just go away.  But in fact, progress on tagging has stalled because we don't know how to distinguish differences of opinions from errors.  When two annotators return two different answers, it is a difference of opinion.  But when a machine returns a different answer, the machine is almost always wrong.  This reader got stuck on the term: adversary.  I think the NIPS paper used that because it was modeling noise under  murphy's law.   It is often wise to assume the worst.  But I don't think it is helpful to think of differences of opinion as an adversarial game like chess.  In chess, it makes sense to think that your opponent is out to get you, but I'm not sure that's the most helpful way to think about differences of opinion.  I think it would clarify what you are doing to say that you are applying an established method from NIPS (that uses the term  adversarial ) to deal with the elusive gold standard problem.  And then point out that the elusive gold standard problem is a very common problem.  You will study it in the context of a particular problem in Chinese, but the problem is much more general than that.  - General Discussion:  I found much of the paper unnecessarily hard going.  I'm not up on Chinese or the latest in NIPS, which doesn't help.  But even so, there are some small issues with English, and some larger problems with exposition.  Consider Table 4.  Line 525 makes an assertion about the first block and depth of networks.  Specifically, which lines in Table 4 support that assertion.  I assume that P and R refer to precision and recall, but where is that explained.  I assume that F is the standard F measure, and OOV is out-of-vocabulary, but again, I shouldn't have to assume such things.  There are many numbers in Table 4.  What counts as significance?  Which numbers are even comparable?  Can we compare numbers across cols?  Is performance on one collection comparable to performance on another?  Line 560 suggests that the adversarial method is not significant.  What should I take away from Table 4?  Line 794 claims that you have a significant solution to what I call the elusive gold standard problem.              But which numbers in Table 4 justify that claim?  Small quibbles about English:  works -to work (in many places).  Work is a  mass noun, not a count noun (unlike  conclusion ).              One can say one conclusion, two conclusions, but more less some work (not one work, two works).  line 493: each dataset, not each datasets  line 485: Three datasets use traditional Chinese (AS, CITY, CKIP) and the other five use simplified Chinese.  line 509: random -to randomize


Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling
Recurrent neural networks (RNNs) have shown promising performance for language modeling. However, traditional training of RNNs using back-propagation through time often suffers from overfitting. One reason for this is that stochastic optimization (used for large training sets) does not provide good estimates of model uncertainty. This paper leverages recent advances in stochastic gradient Markov Chain Monte Carlo (also appropriate for large training sets) to learn weight uncertainty in RNNs. It yields a principled Bayesian learning algorithm, adding gradient noise during training (enhancing exploration of the model-parameter space) and model averaging when testing. Extensive experiments on various RNN models and across a broad range of applications demonstrate the superiority of the proposed approach relative to stochastic optimization.
1 Introduction
Language modeling is a fundamental task, used for example to predict the next word or character in a text sequence given the context. Recently, recurrent neural networks (RNNs) have shown promising performance on this task (Mikolov et al., 2010; Sutskever et al., 2011). RNNs with Long Short-Term Memory (LSTM) units (Hochreiter and Schmidhuber, 1997) have emerged as a popular architecture, due to their representational power and effectiveness at capturing long-term dependencies.
RNNs are usually trained via back-propagation through time (Werbos, 1990), using stochastic optimization methods such as stochastic gradient de-
scent (SGD) (Robbins and Monro, 1951); stochastic methods of this type are particularly important for training with large data sets. However, this approach often provides a maximum a posteriori (MAP) estimate of model parameters. The MAP solution is a single point estimate, ignoring weight uncertainty (Blundell et al., 2015; HernándezLobato and Adams, 2015). Natural language often exhibits significant variability, and hence such a point estimate may make over-confident predictions on test data.
To alleviate overfitting RNNs, good regularization is known as a key factor to successful applications. In the neural network literature, Bayesian learning has been proposed as a principled method to impose regularization and incorporate model uncertainty (MacKay, 1992; Neal, 1995), by imposing prior distributions on model parameters. Due to the intractability of posterior distributions in neural networks, Hamiltonian Monte Carlo (HMC) (Neal, 1995) has been used to provide sample-based approximations to the true posterior. Despite the elegant theoretical property of asymptotic convergence to the true posterior, HMC and other conventional Markov Chain Monte Carlo methods are not scalable to large training sets.
This paper seeks to scale up Bayesian learning of RNNs to meet the challenge of the increasing amount of “big” sequential data in natural language processing, leveraging recent advances in stochastic gradient Markov Chain Monte Carlo (SG-MCMC) algorithms (Welling and Teh, 2011; Chen et al., 2014; Ding et al., 2014; Li et al., 2016a). Specifically, instead of training a single network, SG-MCMC is employed to train an ensemble of networks, where each network has its parameters drawn from a shared posterior distribution. This is implemented by adding additional gradient noise during training and utilizing model
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Encoding weights
Recurrent weights
Decoding weights
Output
Input
Hidden
Figure 1: Illustration of different weight learning strategies in a single-hidden-layer RNN. Stochastic optimization used for MAP estimation puts fixed values on all weights. Naive dropout is allowed to put weight uncertainty only on encoding and decoding weights, and fixed values on recurrent weights. The proposed SG-MCMC scheme imposes distributions on all weights.
averaging when testing. This simple procedure has the following salutary properties for training neural networks: (i) When training, the injected noise encourages model-parameter trajectories to better explore the parameter space. This procedure was also empirically found effective in (Neelakantan et al., 2016). (ii) Model averaging when testing alleviates overfitting and hence improves generalization, transferring uncertainty in the learned model parameters to subsequent prediction. (iii) In theory, both asymptotic and non-asymptotic consistency properties of SG-MCMC methods in posterior estimation have been recently established to guarantee convergence (Chen et al., 2015a; Teh et al., 2016). (iv) SG-MCMC is scalable; it shares the same level of computational cost as SGD in training, by only requiring the evaluation of gradients on a small mini-batch. To the authors’ knowledge, RNN training using SG-MCMC has not been investigated previously, and is a contribution of this paper. We also perform extensive experiments on several natural language processing tasks, demonstrating the effectiveness of SG-MCMC for RNNs, including character/word-level language modeling, image captioning and sentence classification.
2 Related Work
Several scalable Bayesian learning methods have been proposed recently for neural networks. These come in two broad categories: stochastic variational inference (Graves, 2011; Blundell et al., 2015; Hernández-Lobato and Adams, 2015) and SG-MCMC methods (Korattikara et al., 2015; Li et al., 2016a). While prior work focuses on feed-forward neural networks, there has been little if any research reported for RNNs using SG-
MCMC.
Dropout (Hinton et al., 2012; Srivastava et al., 2014) is a commonly used regularization method for training neural networks. Recently, several works have studied how to apply dropout to RNNs (Pachitariu and Sahani, 2013; Bayer et al., 2013; Pham et al., 2014; Zaremba et al., 2014; Bluche et al., 2015; Moon et al., 2015; Semeniuta et al., 2016; Gal and Ghahramani, 2016b). Among them, naive dropout (Zaremba et al., 2014) can impose weight uncertainty only on encoding weights (those that connect input to hidden units) and decoding weights (those that connect hidden units to output), but not the recurrent weights (those that connect consecutive hidden states). It has been concluded that noise added in the recurrent connections leads to model instabilities, hence disrupting the RNN’s ability to model sequences.
Dropout has been recently shown to be a variational approximation technique in Bayesian learning (Gal and Ghahramani, 2016a; Kingma et al., 2015). Based on this, (Gal and Ghahramani, 2016b) proposed a new variant of dropout that can be successfully applied to recurrent layers, where the same dropout masks are shared along time for encoding, decoding and recurrent weights, respectively. Alternatively, we focus on SG-MCMC, which can be viewed as the Bayesian interpretation of dropout from the perspective of posterior sampling (Li et al., 2016b); this also allows imposition of model uncertainty on recurrent layers, enhancing performance. A comparison of naive dropout and SG-MCMC is illustrated in Fig. 1.
3 Recurrent Neural Networks
3.1 RNN as Bayesian Predictive Models
Consider data D = {D1, · · · ,DN}, where Dn , (Xn,Yn), with input Xn and output Yn. Our goal is to learn model parameters θ to best characterize the relationship from Xn to Yn, with corresponding data likelihood p(D|θ) =∏N n=1 p(Dn|θ). In Bayesian statistics, one sets a prior on θ via distribution p(θ). The posterior p(θ|D) ∝ p(θ)p(D|θ) reflects the belief concerning the model parameter distribution after observing the data. Given a test input X̃ (with missing output Ỹ), the uncertainty learned in training is transferred to prediction, yielding the posterior
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
predictive distribution:
p(Ỹ|X̃,D)= ∫ θ p(Ỹ|X̃,θ)p(θ|D)dθ . (1)
When the input is a sequence, RNNs may be used to parameterize the input-output relationship. Specifically, consider input sequence X = {x1, . . . ,xT }, where xt is the input data vector at time t. There is a corresponding hidden state vector ht at each time t, obtained by recursively applying the transition function ht = H(ht−1,xt) (specified in Section 3.2; see Fig. 1). The ouput Y differs depending on the application: a sequence {y1, . . . ,yT } in language modeling or a discrete label in sentence classification. In RNNs the corresponding decoding function is p(y|h), described in Section 3.3.
3.2 RNN Architectures
The transition function H(·) can be implemented with a gated activation function, such as Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) or a Gated Recurrent Unit (GRU) (Cho et al., 2014). Both the LSTM and GRU have been proposed to address the issue of learning long-term sequential dependencies.
Long Short-Term Memory The LSTM architecture addresses the problem of learning longterm dependencies by introducing a memory cell, that is able to preserve the state over long periods of time. Specifically, each LSTM unit has a cell containing a state ct at time t. This cell can be viewed as a memory unit. Reading or writing the cell is controlled through sigmoid gates: input gate it, forget gate ft, and output gate ot. The hidden units ht are updated as
it = σ(Wixt +Uiht−1 + bi) ,
ft = σ(Wfxt +Ufht−1 + bf ) ,
ot = σ(Woxt +Uoht−1 + bo) ,
c̃t = tanh(Wcxt +Ucht−1 + bc) ,
ct = ft ct−1 + it c̃t , ht = ot tanh(ct) ,
where σ(·) denotes the logistic sigmoid function, and represents the element-wise matrix multiplication operator. W{i,f,o,c} are encoding weights, and U{i,f,o,c} are recurrent weights, as shown in Fig. 1. b{i,f,o,c} are bias terms.
Variants Similar to the LSTM unit, the GRU also has gating units that modulate the flow of information inside the hidden unit. It has been shown that a GRU can achieve similar performance to an LSTM in sequence modeling (Chung et al., 2014). We specify the GRU in the Supplementary Material.
The LSTM can be extended to the bidirectional LSTM and multilayer LSTM. A bidirectional LSTM consists of two LSTMs that are run in parallel: one on the input sequence and the other on the reverse of the input sequence. At each time step, the hidden state of the bidirectional LSTM is the concatenation of the forward and backward hidden states. In multilayer LSTMs, the hidden state of an LSTM unit in layer ` is used as input to the LSTM unit in layer `+1 at the same time step.
3.3 Applications
The proposed Bayesian framework can be applied to any RNN model; we focus on the following tasks to demonstrate the ideas.
Language Modeling In word-level language modeling, the input to the network is a sequence of words, and the network is trained to predict the next word in the sequence with a softmax classifier. Specifically, for a length-T sequence, denote yt = xt+1 for t = 1, . . . , T − 1. x1 and yT are always set to a special START and END token, respectively. At each time t, there is a decoding function p(yt|ht) = softmax(Vht) to compute the distribution over words, where V are the decoding weights (the number of rows of V corresponds to the number of words/characters). We also extend this basic language model to consider other applications: (i) a character-level language model can be specified in a similar manner by replacing words with characters (Karpathy et al., 2016). (ii) Image captioning can be considered as a conditional language modeling problem, in which we learn a generative language model of the caption conditioned on an image (Vinyals et al., 2015).
Sentence Classification Sentence classification aims to assign a semantic category label y to a whole sentence X. This is usually implemented through applying the decoding function once at the end of sequence: p(y|hT ) = softmax(VhT ), where the final hidden state of a RNN hT is often considered as the summary of the sentence (here
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
the number of rows of V corresponds to the number of classes).
4 Scalable Learning with SG-MCMC
4.1 The Pitfall of Stochastic Optimization
Typically there is no closed-form solution for the posterior p(θ|D), and traditional Markov Chain Monte Carlo (MCMC) methods (Neal, 1995) scale poorly for largeN . To ease the computational burden, stochastic optimization is often employed to find the MAP solution. This is equivalent to minimizing an objective of regularized loss function U(θ) that corresponds to a (non-convex) model of interest: θMAP = argminU(θ), U(θ) = − log p(θ|D). The expectation in (1) is approximated as:
p(Ỹ|X̃,D)= p(Ỹ|X̃,θMAP) . (2)
Though simple and effective, this procedure largely loses the benefit of the Bayesian approach, because the uncertainty on weights is ignored. To more accurately approximate (1), we employ stochastic gradient (SG) MCMC (Welling and Teh, 2011).
4.2 Large-scale Bayesian Learning
The negative log-posterior is
U(θ) , − log p(θ)− N∑ n=1 log p(Dn|θ). (3)
In optimization,E = − ∑N
n=1 log p(Dn|θ) is typically referred to as the loss function, and R ∝ − log p(θ) as a regularizer.
For large N , stochastic approximations are often employed:
Ũt(θ),− log p(θ)− N
M M∑ m=1 log p(Dim |θ), (4)
where Sm = {i1, · · · , iM} is a random subset of the set {1, 2, · · · , N}, with M N . The gradient on this mini-batch is denoted as f̃t = ∇Ũt(θ), which is an unbiased estimate of the true gradient. The evaluation of (4) is cheap even when N is large, allowing one to efficiently collect a sufficient number of samples in large-scale Bayesian learning, {θs}Ss=1, where S is the number of samples (this will be specified later). These samples are used to construct a sample-based estimation to the expectation in (1):
The finite-time estimation errors of SG-MCMC methods are bounded (Chen et al., 2015a), which guarantees (5) is an unbiased estimate of (1) asymptotically under appropriate decreasing stepsizes.
4.3 SG-MCMC Algorithms
SG-MCMC and stochastic optimization are parallel lines of work, designed for different purposes; their relationship has recently been revealed in the context of deep learning. The most basic SG-MCMC algorithm has been applied to Langevin dynamics, and is termed SGLD (Welling and Teh, 2011). To help convergence, a momentum term has been introduced in SGHMC (Chen et al., 2014), a “thermostat” has been devised in SGNHT (Ding et al., 2014) and preconditioners have been employed in pSGLD (Li et al., 2016a). These SG-MCMC algorithms often share similar characteristics with their counterpart approaches from the optimization literature such as the momentum SGD, Santa (Chen et al., 2016) and RMSprop/Adagrad (Tieleman and Hinton, 2012; Duchi et al., 2011). The interrelationships between SG-MCMC and optimization-based approaches are summarized in Table 1.
SGLD Stochastic Gradient Langevin Dynamics (SGLD) (Welling and Teh, 2011) draws posterior samples, with updates
θt = θt−1 − ηtf̃t−1 + √ 2ηtξt , (6)
where ηt is the learning rate, and ξt ∼ N (0, Ip) is a standard Gaussian random vector. SGLD is the SG-MCMC analog to stochastic gradient descent (SGD), whose parameter updates are given by:
θt = θt−1 − ηtf̃t−1 . (7)
SGD is guaranteed to converge to a local minimum under mild conditions (Bottou, 2010). The
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Algorithm 1: pSGLD Input: Default hyperparameter settings:
ηt = 1×10−3, λ = 10−8, β1 = 0.99. Initialize: v0 ← 0, θ1 ∼ N (0, I) ; for t = 1, 2, . . . , T do
% Estimate gradient from minibatch St f̃t = ∇Ũt(θ); % Preconditioning vt ← β1vt−1 + (1− β1)f̃t f̃t;
G−1t ← diag ( 1 ( λ1+ v 1 2 t )) ;
% Parameter update ξt ∼ N (0, ηtG−1t ); θt+1← θt + ηt2 G −1 t f̃t+ ξt;
end
additional Gaussian term in SGLD helps the learning trajectory to explore the parameter space to approximate posterior samples, instead of obtaining a local minimum.
pSGLD Preconditioned SGLD (pSGLD) (Li et al., 2016a) was proposed recently to improve the mixing of SGLD. It utilizes magnitudes of recent gradients to construct a diagonal preconditioner to approximate the Fisher information matrix, and thus adjusts to the local geometry of parameter space by equalizing the gradients so that a constant stepsize is adequate for all dimensions. This is important for RNNs, whose parameter space often exhibits pathological curvature and saddle points (Pascanu et al., 2013), resulting in slow mixing. There are multiple choices of preconditioners; similar ideas in optimization include Adagrad (Duchi et al., 2011), Adam (Kingma and Ba, 2015) and RMSprop (Tieleman and Hinton, 2012). An efficient version of pSGLD, adopting RMSprop as the preconditioner G, is summarized in Algorithm 1, where denotes elementwise matrix division. When the preconditioner is fixed as the identity matrix, the method reduces to SGLD.
4.4 Understanding SG-MCMC
To further understand SG-MCMC, we show its close connection to dropout/dropConnect (Srivastava et al., 2014; Wan et al., 2013). These methods improve the generalization ability of deep models, by randomly adding binary/Gaussian noise to the local units or global weights. For neural networks with the nonlinear function q(·) and consecutive
layers h1 and h2, dropout and dropConnect are denoted as:
Dropout: h2 = ξ0 q(θh1), DropConnect: h2 = q((ξ0 θ)h1),
where the injected noise ξ0 can be binary-valued with dropping rate p or its equivalent Gaussian form (Wang and Manning, 2013):
Binary noise: ξ0 ∼ Ber(p),
Gaussian noise: ξ0 ∼ N (1, p
1− p ).
Note that ξ0 is defined as a vector for dropout, and a matrix for dropConnect. By combining dropConnect and Gaussian noise from the above, we have the update rule (Li et al., 2016b):
θt+1 = ξ0 θt − η
2 f̃t = θt −
η 2 f̃t + ξ ′ 0 , (8)
where ξ′0 ∼ N ( 0, p(1−p)diag(θ 2 t ) )
; (8) shows that dropout/ dropConnect and SGLD in (6) share the same form of update rule, with the distinction being that the level of injected noise is different. In practice, the noise injected by SGLD may not be enough. A better way that we find to improve the performance is to jointly apply SGLD and dropout. This method can be interpreted as using SGLD to sample the posterior distribution of a mixture of RNNs, with mixture probability controlled by the dropout rate.
5 Experiments
We present results on several tasks, including character/ word-level language modeling, image captioning, and sentence classification. The hyperparameter settings, the initialization of model parameters and model specifications on each dataset are provided in the Supplementary Material.
We do not perform any dataset-specific tuning other than early stopping on validation sets. When dropout is utilized, the dropout rate is set to 0.5. All experiments are implemented in Theano (Theano Development Team, 2016), using a NVIDIA GeForce GTX TITAN X GPU with 12GB memory.
5.1 Language Modeling
We first test character-level and word-level language modeling. The setup for each task is as follows.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
• Following (Karpathy et al., 2016), we test character-level language modeling on the War and Peace (WP) novel. The training/validation/test sets contain 260/32/33 batches, in which there are 100 characters. The vocabulary size is 87, and we consider a 2-hidden-layer RNN of dimension 128.
• The Penn Treebank (PTB) corpus (Marcus et al., 1993) is used for word-level language modeling. The dataset adopts the standard split (929K training words, 73K validation words, and 82K test words) and has a vocabulary of size 10K. We train LSTMs of three sizes; these are denoted the small/medium/large LSTM. All LSTMs have two layers and are unrolled for 20 steps. The small, medium and large LSTM has 200, 650 and 1500 units per layer, respectively.
We study the effects of different types of architecture (LSTM/GRU/Vanilla RNN (Karpathy et al., 2016)) on the WP dataset, and effects of different learning algorithms on the PTB dataset. The comparison of test cross-entropy loss on WP is shown in Table 2. We observe that pSGLD consistently outperforms RMSprop. Table 3 summarizes the test set performance on PTB1. It is clear that our sampling-based method consistently outperforms the optimization counterpart, where the performance gain mainly comes from adding gradient noise and model averaging. When compared with dropout, SGLD performs better on the small LSTM model, but slightly worse on the medium and large LSTM model. This may imply that dropout is suitable to regularizing large networks, while SGLD exhibits better regularization ability on small networks, partially due to the fact that dropout may inject a higher level of noise during training than SGLD. In order to inject a higher level of noise into SGLD, we empirically apply SGLD and dropout jointly, and found that this provided the best performace on the medium and large LSTM model.
We study three strategies to do model averaging, i.e., forward collection, backward collection and thinned collection. Given samples (θ1, · · · ,θK) and the number of samples S used for averaging, forward collection refers to using (θ1, · · · ,θS) for
1The results reported here do not match (Zaremba et al., 2014) due to the differences of experimental setup. However, we provide a fair comparison to all methods.
Methods LSTM GRU RNN RMSprop 1.3607 1.2759 1.4239 pSGLD 1.3375 1.2561 1.4093
Table 3: Test perplexity on Penn Treebank.
Methods Small Medium Large SGD 123.85 126.31 130.25 SGD+Dropout 136.39 100.12 97.65 SGLD 117.36 109.14 105.86 SGLD+Dropout 139.54 99.58 94.03
10 20 30 40 50 60 Individual Sample
110 120 130 140 150 160 170 180 Pe rp le xi
ty 0 10 20 30 40 50 60 Number of Samples for Model Averaging
110 120 130 140 150 160 170 180 Pe rp le xi ty
forward collection backward collection thinned collection
(a) Single sample (b) Different collections
Figure 2: Effects of collected samples.
the evaluation of a test function, backward collection refers to using (θK−S+1, · · · ,θK), while thinned collection chooses samples from θ1 to θK with interval K/S. Fig. 2 plots the effects of these strategies, where Fig. 2(a) plots the perplexity of every single sample, Fig. 2(b) plots the perplexities using the three schemes. Only after 20 samples is a converged perplexity achieved in the thinned collection, while it requires 30 samples for forward collection or 60 samples for backward collection. This is unsurprising, because thinned collection provides a better way to select samples. Nevertheless, averaging of samples provides significantly lower perplexity than using single samples. Note that the overfitting problem in Fig. 2(a) is also alleviated by model averaging.
To better illustrate the benefit of model averaging, we visualize in Fig. 3 the probabilities of each word in a randomly chosen test sentence. The first 3 rows are the results predicted by 3 distinctive model samples, respectively; the bottom row is the result after averaging. Their corresponding perplexities for the test sentence are also shown on the right of each row. The 3 individual samples provide reasonable probabilities. For example, the consecutive words “New York”, “stock exchange” and “did not” are assigned with a higher probability. After averaging, we can see a much lower perplexity, as the samples can complement each other. For example, though the second sample can
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Methods B-1 B-2 B-3 B-4 METEOR CIDEr ROUGE-L Perp. Results on Flickr8k RMSprop 0.640 0.427 0.288 0.197 0.205 0.476 0.500 16.64 RMSprop + Dropout 0.647 0.444 0.305 0.209 0.208 0.514 0.510 15.72 pSGLD 0.669 0.463 0.321 0.224 0.214 0.535 0.522 14.29 pSGLD + Dropout 0.656 0.450 0.309 0.211 0.209 0.512 0.512 14.26 Results on Flickr30k RMSprop 0.644 0.422 0.279 0.184 0.180 0.372 0.476 17.80 RMSprop + Dropout 0.656 0.435 0.295 0.200 0.185 0.396 0.481 18.05 pSGLD 0.657 0.438 0.300 0.206 0.192 0.421 0.490 15.61 pSGLD + Dropout 0.666 0.448 0.308 0.209 0.189 0.419 0.487 17.05
25.55the new york stock exchange did not fall apart 22.24the new york stock exchange did not fall apart 29.83the new york stock exchange did not fall apart
21.98the new york stock exchange did not fall apart 0
0.2
0.4
0.6
0.8
1
Figure 3: Predictive probabilities obtained by 3 samples and their average. Colors indicate normalized probability of each word. Best viewed in color.
a"tan"dog"is"playing"in"the"grass a"tan"dog"is"playing"with"a"red"ball"in"the"grass a"tan"dog"with"a"red"collar"is"running"in"the"grass
a"yellow"dog"runs"through"the"grass a"yellow"dog"is"running"through"the"grass a"brown"dog"is"running"through"the"grass
a"group"of"people"stand"in"front"of"a"building a"group"of"people"stand"in"front"of"a"white"building a"group"of"people"stand"in"front"of"a"large"building
a"man"and"a"woman"walking"on"a"sidewalk a"man"and"a"woman"stand"on"a"balcony a"man"and"a"woman"standing"on"the"ground
Figure 4: Image captioning with different samples. Left are the given images, right are the corresponding captions. The captions in each box are from the same model sample.
yield the lowest single-model perplexity, its prediction on word “York” is still benefited from the other two via averaging.
5.2 Image Caption Generation
We next consider the problem of image caption generation, which is a conditional RNN model, where image features are extracted by residual network (He et al., 2016), and then fed into the RNN to generate the caption. We present results on two benchmark datasets, Flickr8k (Hodosh et al., 2013) and Flickr30k (Young et al., 2014). These datasets contain 8,000 and 31,000 images, respectively. Each image is annotated with 5 sentences. A single-layer LSTM is employed with the number of hidden units set to 512.
The widely used BLEU (Papineni et al., 2002),
METEOR (Banerjee and Lavie, 2005), ROUGEL (Lin, 2004), and CIDEr-D (Vedantam et al., 2015) metrics are used to evaluate the performance. All the metrics are computed by using the code released by the COCO evaluation server (Chen et al., 2015b).
Table 4 presents results for pSGLD/RMSprop with or without dropout. Consistent with the results in the basic language modeling, pSGLD yields improved performance compared to RMSprop. For example, pSGLD provides 2.77 BLEU-4 score improvement over RMSprop on the Flickr8k dataset. By comparing pSGLD with RMSprop with dropout, we conclude that pSGLD exhibits better regularization ability than dropout on these two datasets.
Apart from modeling weight uncertainty, different samples from our algorithm may capture different aspects of the input image. An example with two images is shown in Fig. 4, where 2 randomly chosen model samples are considered for each image. For each model sample, the top 3 generated captions are presented. We use the beam search approach (Vinyals et al., 2015) to generate captions, with a beam of size 5. In Fig. 4, the two samples for the first image mainly differ in the color and activity of the dog, e.g., “tan” or “yellow”, “playing” or “running”, whereas for the second image, the two samples reflect different understanding of the image content.
5.3 Sentence Classification
We study the task of sentence classification on 5 datasets as in (Kiros et al., 2015): TREC (Li and Roth, 2002), MR (Pang and Lee, 2005), SUBJ (Pang and Lee, 2004), CR (Hu and Liu, 2004) and MPQA (Wiebe et al., 2005). A singlelayer bidirectional LSTM is employed with the
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Table 5: Sentence classification errors on five benchmark datasets.
Methods MR CR SUBJ MPQA TREC RMSprop 21.26±1.45 22.70±2.20 8.13±1.19 10.60±1.28 8.58±0.79 RMSprop + Dropout 20.33±0.67 20.70±2.22 7.24±0.86 10.66±0.74 7.82±0.66 RMSprop + Gal’s Dropout 19.66±0.60 20.21±2.34 7.52±1.17 10.59±1.12 8.32±0.52 pSGLD 20.36±1.09 21.18±1.90 7.43±1.21 10.54±0.99 7.88±0.72 pSGLD + Dropout 19.48±0.95 19.74±2.03 6.61±1.06 10.22±0.89 7.32±0.66
5 10 15 #Epoch
0.00
0.05
0.10
0.15
0.20
0.25
Er ro
r
Train RMSprop RMSprop + Dropout pSGLD pSGLD + Dropout
5 10 15 #Epoch
0.10
0.12
0.14
0.16
0.18
0.20
0.22
0.24
0.26
Er ro
r
Validation
5 10 15 #Epoch
0.10
0.15
0.20
Er ro
r
Test
Figure 5: Learning curves on TREC dataset.
number of hidden units set to 400. Table 5 shows the testing classification errors. 10-fold cross-validation is used for evaluation on the first 4 datasets, while TREC has a pre-defined training/test split, and we run each algorithm 10 times on TREC. In addition to (naive) dropout, we further compare pSGLD with the Gal’s dropout, recently proposed in (Gal and Ghahramani, 2016b), which is shown to be applicable to recurrent layers. The combination of pSGLD and dropout consistently provides the lowest errors.
In the following, we focus on the analysis of TREC. Each sentence of TREC is a question, and the goal is to decide which topic type the question is most related to: location, human, numeric, abbreviation, entity or description. Fig. 5 plots the learning curves of different algorithms on the training, validation and testing sets of the TREC dataset. pSGLD and dropout have similar behavior: they explore the parameter space during learning, and thus coverge slower than RSMprop on the training dataset. However, the learned uncertainty alleviates overfitting and results in lower errors on the validation and testing datasets.
To further study the Bayesian nature of the proposed approach, in Fig. 6 we choose two testing sentences with high uncertainty (i.e., standard derivation in prediction) from the TREC dataset. Interestingly, after embedding to 2d-space with tSNE (Van der Maaten and Hinton, 2008), the two sentences correspond to points lying on the boundary of different classes. We use 20 model sam-
Whatdoes cc in engines mean?
Whatdoes adefibrillatordo?
True5Type Predicted5 Type
Description
Description
Testing5Question
Entity
Abbreviation
Figure 6: Visualization. Top two rows show selected ambiguous sentences, which correspond to the points with black circles in tSNE visualization of the testing dataset.
ples to estimate the prediction mean and standard derivation on the true type and predicted type. The classifier yields higher probability on the wrong types, associated with higher standard derivations. One can leverage the uncertainty information to make decisions: either manually make a human judgement when uncertainty is high, or automatically choose the one with lower standard derivations when both types exhibits similar prediction means. A more rigorous usage of the uncertainty information is left as future work.
6 Conclusion
We propose a scalable Bayesian learning framework using SG-MCMC, to model weight uncertainty in recurrent neural networks. The learning framework is tested on several tasks, including language models, image caption generation and sentence classification. Our algorithm outperforms stochastic optimization algorithms, indicating the importance of learning weight uncertainty in recurrent neural networks. Our algorithm requires little additional computational overhead in training, and multiple times of forward-passing for model averaging in testing. Future works include improving the testing efficiency for the large-scale RNNs, via learning a single neural network that approximates the model averaging result (Korattikara et al., 2015).
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths:  a) The paper presents a Bayesian learning approach for recurrent neural network language model. The method outperforms standard SGD with dropout on three tasks.  b) The idea of using Bayesian learning with RNNs appears to be novel.  c) The computationally efficient Bayesian algorithm for RNN would be of interest to the NLP community for various applications.  - Weaknesses:  Primary concern is about evaluation:  Sec 5.1: The paper reports the performance of difference types of architectures (LSTM GRU vanilla RNN) on character LM task while comparing the learning algorithms on the Penn Treebank task. Furthermore, RMSprop and pSGLD are compared for the character LM while SGD + - dropout is compared with SGLD + - dropout on word language model task. This is inconsistent!  I would suggest reporting both these dimensions (i.e. architectures and the exact same learning algorithms) on both character and word LM tasks. It would be useful to know if the results from the proposed Bayesian learning approaches are portable across both these tasks and data sets.  L529: The paper states that 'the performance gain mainly comes from adding gradient noise and model averaging'. This statement is not justified empirically. To arrive at this conclusion, an A B experiment with without adding gradient noise and or model averaging needs to be done.   L724: Gal's dropout is run on the sentence classification task but not on language model captions task. Since Gal's dropout is not specific to sentence classification,  I would suggest reporting the performance of this method on all three tasks. This would allow the readers to fully assess the utility of the proposed algorithms relative to all existing dropout approaches.  L544: Is there any sort order for the samples? (  theta_1, ...,   theta_K)? e.g. are samples with higher posterior probabilities likely to be at higher indices? Why not report the result of randomly selecting K out of S samples, as an additional alternative?  Regular RNN LMs are known to be expensive to train and evaluate. It would be very useful to compare the training evaluation times for the proposed Bayesian learning algorithms with SGD+ dropout. That would allow the readers to trade-off improvements versus increase in training run times.  Clarifications: L346: What does   theta_s refer to? Is this a MAP estimate of parameters based on only the sample s? L453-454: Clarify what   theta means in the context of dropout dropconnect.   Typos: L211: output L738: RMSProp