Handling Cold-Start Problem in Review Spam Detection by Jointly Embedding Texts and Behaviors
Solving cold-start problem in review spam detection is an urgent and significant task. It can help the on-line review websites to relieve the damage of spammers in time, but has never been investigated by previous work. This paper proposes a novel neural network model to detect review spam for cold-start problem, by learning to represent the new reviewers’ review with jointly embedded textual and behavioral information. Experimental results prove the proposed model achieves an effective performance and possesses preferable domain-adaptability. It is also applicable to a large scale dataset in an unsupervised way.
1 Introduction
With the rapid growth of products reviews at the web, it has become common for people to read reviews before making purchase decision. The reviews usually contain abundant consumers’ personal experiences. It has led to a significant influence on financial gains and fame for businesses. Existing studies have shown that an extra halfstar rating on Yelp causes restaurants to sell out 19% points more frequently (Anderson and Magruder, 2012), and a one-star increase in Yelp rating leads to a 5-9 % increase in revenue (Luca, 2011). This, unfortunately, gives strong incentives for imposters (called spammers) to game the system. They post fake reviews or opinions (called review spam) to promote or to discredit some targeted products and services. The news from BBC has shown that around 25% of Yelp reviews could be fake.1 Therefore, it is urgent to detect review s-
1http://www.bbc.com/news/technology-24299742
pam, to ensure that the online review continues to be trusted.
Jindal and Liu (2008) make the first step to detect review spam. Most efforts are devoted to explore effective linguistic and behavioral features by subsequent work to distinguish such spam from the real reviews. However, to notice such patterns or form behavioral features, developers should take long time to observe the data, because the features are based on statistics. For instance, the feature activity window proposed by Mukherjee et al. (2013c) is to measure the activity freshness of reviewers. It usually takes several months to count the difference of timestamps between the last and first reviews for reviewers. When the features show themselves finally, some major damages might have already been done. Thus, it is important to design algorithms that can detect review spam as soon as possible, ideally, right after they are posted by the new reviewers. It is a coldstart problem which is the focus of this paper.
In this paper, we assume that we must identify fake reviews immediately when a new reviewer posts just one review. Unfortunately, it is very difficult because the available information for detecting fake reviews is very poor. Traditional behavioral features based on the statistics can only work well on users’ abundant behaviors. The more behavioral information obtained, the more effective the traditional behavioral features are (see experiments in Section 3 ). In the scenario of cold-start, a new reviewer only has a behavior: post a review. As a result, we can not get effective behavioral features from the data. Although, the linguistic features of reviews do not need to take much time to form, Mukherjee et al. (2013c) have proved that the linguistic features are not effective enough in detecting real-life fake reviews from the commercial websites, where we also obtain the same observation (the details are shown in Section 3).
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Therefore, the main difficulty of the cold-start spam problem is that there is no sufficient behaviors of the new reviewers for constructing effective behavioral features. Nevertheless, there are ample textual and behavioral information contained in the abundant reviews posted by the existing reviewers (Figure 1). We could employ behavioral information of existing similar reviewers to a new reviewer to approximate his behavioral features. We argue that a reviewer’s individual characteristics such as background information, motivation and interactive behavior style have a great influence on a reviewer’s textual and behavioral information. So the textual information and the behavioral information of a reviewer are correlated with each other (similar argument in Li et al. (2016)). For example, the students of college are likely to choose the youth hostel during summer vacation, and tend to comment the room price in their reviews. But the financial analysts on business trip may tend to choose the business hotel, the environment and service are what they care about in their reviews.
To augment the behavioral information of the new reviewers in the cold-start problem, we first try to find the textual information which is similar with that of the new reviewer, from the existing reviews. There are several ways to model the textual information of the review spam, such as Unigram (Mukherjee et al., 2013c), POS (Ott et al., 2011) and LIWC (Linguistic Inquiry and Word Count) (Newman et al., 2003). We employ the CNN (Convolutional Neural Network) to model the review text, which has been proved that it can capture complex global semantic information that is difficult to express using traditional discrete manual features (Ren and Zhang, 2016). Then we employ the behavioral information which is correlated with the found textual information to approximate the behavioral information of the new reviewer. An intuitive approach is to search the most similar existing review for the new review, then take the found reviewer’s behavioral features as the new reviewers’ features (detailed in Section 5.3). However, there are abundant behavioral information in the review graph (Figure 1), it is difficult for the traditional discrete manual behavioral features to record the global behavioral information (Wang et al., 2016). Moreover, the traditional features can not capture the reviewer’s individual characteristics, because there is no explicit characteristic tag available in the review system (experi-
ments in Section 5.3). So, we propose a neural network model to jointly encode the textual and behavioral information into the review embeddings for detecting the review spam in cold-start problem. By encoding the review graph structure (Figure 1), the proposed model can record the global footprints of the existing reviewers in an unsupervised way, and further record the reviewers’ latent characteristic information in the footprints. The jointly learnt review embeddings can model the correlation of the reviewers’ textual and behavioral information. When a new reviewer posts a review, the proposed model can represent the review with the similar textual information and the correlated behavioral information encoded in the word embeddings. Finally, the embeddings of the new review are fed into a classifier to identify whether it is spam or not.
In summary, our major contributions include: • To our best knowledge, this is the first work
that explores the cold-start problem in review spam detection. We qualitatively and quantitatively prove that the traditional linguistic and behavioral features are not effective enough in detecting review spam for the coldstart task. • We propose a neural network model to joint-
ly encode the textual and behavioral information into the review embeddings for the cold-start spam detection task. It is an unsupervised distributional representation model which can learn from large scale unlabeled review data. • Experimental results on two domains (hotel
and restaurant ) give good confidence that the proposed model performs effectively in the cold-start spam detection task.
2 Related Work
Jindal and Liu (2008) make the first step to detect review spam. Subsequent work devoted most efforts to explore effective features and spammerlike clues.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Linguistic features: Ott et al. (2011) applied psychological and linguistic clues to identify review spam; Harris (2012) explored several writing style features. Syntactic stylometry for review spam detection was investigated in Feng et al. (2012a); Xu and Zhao (2012) using deep linguistic features for finding deceptive opinion spam; Li et al. (2013) studied the topics in the review spam; Li et al. (2014b) further analyzed the general difference of language usage. Fornaciari and Poesio (2014) proved the effectiveness of the N-grams in detecting deceptive Amazon book reviews. The effectiveness of the N-grams was also explored in Cagnina and Rosso (2015). Li et al. (2014a) proposed a positive-unlabeled learning method based on unigrams and bigrams; Kim et al. (2015) carried out a frame-based deep semantic analysis. Hai et al. (2016) exploited the relatedness of multiple review spam detection tasks and available unlabeled data to address the scarcity of labeled opinion spam data by using linguistic features. Besides, (Ren and Zhang, 2016) proved that the CNN model is more effective than the RNN and the traditional discrete manual linguistic features. Hovy (2016) used N-gram generative models to produce reviews and evaluated their effectiveness.
Behavioral features: Lim et al. (2010) analyzed reviewers’ rating behavioral features; Jindal et al. (2010) identified unusual review patterns which can represent suspicious behaviors of reviews; Li et al. (2011) proposed a two-view semisupervised co-training method base on behavioral features. Feng et al. (2012b) study the distributions of individual spammers’ behaviors. The group spammers’ behavioral features were studied in Mukherjee et al. (2012). Temporal patterns of spammers were investigated by Xie et al. (2012), Fei et al. (2013); Li et al. (2015) explored the temporal and spatial patterns. The review graph was analyzed by Wang et al. (2011), Akoglu et al. (2013); Mukherjee et al. (2013a) studied the spamicity of reviewers. Mukherjee et al. (2013c), Mukherjee et al. (2013b) proved that reviewers’ behavioral features are more effective than reviews’ linguistic features for detecting review spam. Based on this conclusion, recently, researchers (Rayana and Akoglu, 2015; KC and Mukherjee, 2016) have put more efforts in employing reviewers’ behavioral features for detecting review spam, the intuition behind which is to capture the reviewers’ actions and supposes that
those reviews written with spammer-like behaviors would be spam. Wang et al. (2016) explored a method to learn the review representation with global behavioral information.
3 Whether Traditional Features are Effective
As a new reviewer posted just one review and we have to identify it immediately, the major challenge of the cold-start task is that, the available informations about the new reviewer are very poor. The new reviewer only provide us with one review record. For most traditional features based on the statistics, they can not form themselves or make no sense, such as the percentage of reviews written at weekends (Li et al., 2015), the entropy of rating distribution of user’s review (Rayana and Akoglu, 2015). To investigate whether traditional features are effective in the cold-start task, we conducted experiments on the Yelp dataset in Mukherjee et al. (2013c). We trained SVM models with different features on the existing reviews posted before January 1, 2012, and tested on the new reviews which just posted by the new reviewers after January 1, 2012. Results are shown in Table 1.
3.1 Linguistic Features’ Poor Performance
The linguistic features need not to take much time to form. But Mukherjee et al. (2013c) have proved that the linguistic features are not effective enough in detecting real-life fake reviews from the commercial websites, compared with the performances on the crowd source datasets (Ott et al.,
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
391
392
393
394
395
396
397
398
399
2011). They showed that the word bigrams perform better than the other linguistic features, such as LIWC (Newman et al., 2003; Pennebaker et al., 2007), part-of-speech sequence patterns (Mukherjee and Liu, 2010), deep syntax (Feng et al., 2012a), information gain (Mukherjee et al., 2013c) and so on. So, we conduct experiments with the word bigrams feature. As shown in Table 1 (a, b) row 1, the word bigrams result in only around 55% in accuracy in both the hotel and restaurant domains. It indicates that the most effective traditional linguistic feature (i.e., the word bigrams) can’t detect the review spam effectively in the cold start task.
3.2 Behavioral Features only Work Well with
Abundant Information
Because there is not enough available information about the new reviewer, for most traditional behavioral features based on the statistical mechanism, they couldn’t form themselves or make no sense. We investigated the previous work and found that there are three behavioral features can be applied to the cold-start task. They are proposed by Mukherjee et al. (2013b), i.e., 1.Review length (RL) : the length of the new review posted by the new reviewer; 2.Reviewer deviation (RD): the absolute rating deviation of the new reviewer’s review from other reviews on the same business; 3.Maximum content similarity (MCS) : the maximum content similarity (using cosine similarity) between the new reviewer’s review with other reviews on the same business.
Table 1 (a, b) row 2 shows the experiment results by the combinations of the bigrams feature and the three behavioral features described above. The behavioral features make around 5% improvement in accuracy in the hotel domain (2.7% in the restaurant domain) as compared with only using bigrams. The accuracy is improved but it is just near 60% in average. It indicates that the traditional features are not effective enough with poor behavioral information. What’s more, the behavioral features cause around 4.6% decrease in F1score and around 19% decrease in Recall in both hotel and restaurant domains. It is obvious that there is more false-positive review spam caused by the behavioral features as compared to only using bigrams. It further indicates that the traditional behavioral features’ discrimination for review spam gets to be weakened by the poor behavioral infor-
mation. To go a step further, we carried experiments with the three behavioral features which are formed on abundant behavioral information. When the new reviewers continue to post more reviews in after weeks, their behavioral information gets to be more. Then the review system could obtain more sufficient data to extract behavior features as compared to the poor information in the cold-start period. So the behavioral features with abundant information make an obvious improvement in accuracy (6.4%) in the hotel domain (Table 1 (a) row 3) as compared with the results in Table 1 (a) row 2. But it is only 0.6% in the restaurant domain. By statistics on the datasets, we found that the new reviewers posted about 54.4 reviews in average after their first post in the hotel domain, but it is only 10 reviews in average for the new reviewers in the restaurant domain. The added behavioral information in the hotel domain is richer than that in the restaurant domain. It indicates that:
4 The Proposed Model
The difficulty of detecting review spam in the cold-start task is that the available behavioral information of new reviewers is very poor. The new reviewer just posted one review and we have to filter it out immediately, there is not any historical reviews provided to us. As we argued, the textual in-
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
formation and the behavioral information of a reviewer are correlated with each other. So, to augment the behavioral information of new reviewers, we try to find the textual information which is similar with that of the new reviewer, from existing reviews. Then we take the behavioral information which is correlated with the found textual information as the most possible behavioral information of the new reviewer. For this purpose, we propose a neural network model to jointly encode the textual and behavioral information into the review embeddings for detecting the review spam in the cold-start problem (shown in Figure 2). When a new reviewer posts a review, the neural network can represent the review with the similar textual information and the correlated behavioral information encoded in the word embeddings. Finally, embeddings of the new review are fed into a classifier to identify whether it is spam or not.
4.1 Behavioral Information Encoding
In Figure 1, there is a part of review graph which is simplified from the Yelp website. As it shows, the review graph contains the global behavioral information (footprints) of the existing reviewers. Because the motivations of the spammers and the real reviewers are totally different, the distributions of the behavioral information of them are different (Mukherjee et al., 2013a). There are businesses (even highly reputable ones) paying people to write fake reviews for them to promote their products/services and/or to discredit their competitors (Liu, 2015). So the behavioral footprints of the spammers are decided by the demands of the businesses. But the real reviewers only post reviews to the product or services they have actually experienced. Their behavioral footprints are influenced by their own characteristics. Previous work extracts behavioral features for reviewers from these behavioral information. But it is impractical to the new reviewers in the cold-start task. Moreover, the traditional discrete features can not effectively record the global behavioral information (Wang et al., 2016). Besides, there is no explicit characteristic tag available in the review system, and we need to find a way to record the reviewers’ latent characters information in footprints.
Therefore we encode these behavioral information into our model by utilizing a embedding learning model which is similar with TransE (Bordes et al., 2013). TransE is a model which can encode the graph structure, and represent the nodes
and edges (head, translation/relation, tail) in low dimension vector space. TransE has been proved that it is good at describing the global information of the graph structure by the work about distributional representation for knowledge base (Guu et al., 2015). We consider that each reviewer in review graph describes the product in his/her own view and writes the review. When we represent the product, reviewer and review in low dimension vector space, the reviewer embeddings can be taken as a translation vector, which has translated the product embeddings to the review embeddings. So, as shown in Figure 2, we take the products (hotels/restaurants) as the head part of the TransE network in our model, take the reviewers as the translation (relation) part and take the review as the tail part. By learning from the existing large scale unlabeled reviews of the review graph, we can encode the global behavioral information into our model without extracting any traditional behavioral feature, and record reviewers’ latent characteristics information.
More formally, we minimize a margin-based criterion over the training set:
L = ∑
(β,α,τ )∈S ∑ (β′,α,τ ′)∈S′ max
{0, 1 + d(β +α, τ )− d(β′ +α, τ ′)} (1)
S denotes the training set of triples (β,α, τ ) composed product β (β ∈ B, products set (head part)), reviewer α (α ∈ A, reviewers set (translation part)) and review text embeddings learnt by the CNN τ (τ ∈ T , review texts set (tail part)).
S′ = {(β′,α, τ )|β′ ∈ B} ∪ {(β,α, τ ′)|τ ′ ∈ T} (2)
The set of corrupted triplets S′ (Equation (2)), is composed of training triplets with either the product or review text replaced by a random chosen one (but not both at the same time).
d(β +α, τ ) = ‖β +α− τ‖22 , s.t. ‖β‖22 = ‖α‖ 2 2 = ‖τ‖ 2 2 = 1
(3)
d(β + α, τ ) is the dissimilarity function with the squared euclidean distance.
4.2 Textual Information Encoding
To encode the textual information into our model, we adopt a convolutional neural network (CNN) to learn to represent the existing reviews. By statistics, we find that a review usually refers to several aspects of the products or services. For example, a hotel review may comment the room price, the free
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
556
557
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
WiFi and the bathroom at the same time. Compared with the recurrent neural network (RNN), the CNN can do a better job of modeling the different aspects of a review. Ren and Zhang (2016) have proved that the CNN can capture complex global semantic information and detect review spam more effectively, compared with traditional discrete manual features and the RNN model. As shown in Figure 2, we take the learnt embeddings τ of reviews by the CNN as the tail part.
Specifically, we denote the review text consisting of n words as {w1, w2, ..., wn}, the word embeddings e(wi) ∈ RD, D is the word vector dimension. We take the concatenation of the word embeddings in a fixed length window size Z as the input of the linear layer, which is denoted as Ii ∈ RD×Z . So the output of the linear layer Hi is calculated by Hk,i = Wk · Ii + bi, where Wk ∈ RD×Z is the weight matrix of filter k. We utilize a max pooling layer to get the output of each filter. Then we take tanh as the activation function and concatenate the outputs as the final review embeddings, which is denoted as τi.
4.3 Jointly Information Encoding
To model the correlation of the textual and behavioral information, we employ the jointly information encoding. By jointly learning from the global review graph, the textual and behavioral information of existing spammers and real reviewers are embedded into the word embeddings.
In addition, the rating usually represents the sentiment polarity of a review, e.g., five star means ‘like’ and one star means ‘dislike’. The spammers often review their target products with low rating for discredited purpose, and with high rating for promoted purpose. To encode the semantics of the sentiment polarity into the review embeddings, we learn the embeddings of 1-5 stars rating in our model at the same time. They are taken as the constraints of the review embeddings during the joint learning. They are calculated as:
C = ∑
(τ ,γ)∈Γ ∑ (τ ,γ′)∈Γ′ max{0, 1+ g(τ ,γ)− g(τ ,γ′)} (4)
The set of corrupted tuples Γ′ is composed of training tuples Γ with the rating of review replaced by its opposite rating (i.e., 1 by 5, 2 by 4, 3 by 1 or 5). g(τ ,γ) = ‖τ − γ‖22, norm constraints: ‖γ‖22 = 1.
The final joint loss function is as follows: LJ = (1− θ)L+ θC (5)
where θ is a hyper-parameter.
5 Experiments
5.1 Datasets and Evaluation Metrics
Datasets: To evaluate the proposed method, we conducted experiments on Yelp dataset that was used in (Mukherjee et al., 2013b,c; Rayana and Akoglu, 2015). The statistics of the Yelp dataset are listed in Table 2 and Table 3. The reviewed product here refers to a hotel or restaurant. We take the existing reviews posted before January 1, 2012 as the training datasets, and take the first new reviews which just posted by the new reviewers after January 1, 2012 as the test datasets. Evaluation Metrics: We select precision (P), recall (R), F1-Score (F1), accuracy (A) as metrics.
5.2 Our Model v.s. the Traditional Features
To illustrate the effectiveness of our model, we conduct experiments on the public datasets, and make comparison with the most effective traditional linguistic features, e.g., bigrams, and the three practicable traditional behavioral features (RL, RD, MCS (Mukherjee et al., 2013b)) referred in Section 3.2. The results are shown in Table 4. For our model, we set the dimension of embeddings to 100, the number of CNN filters to 100, θ to 0.1, Z to 2. The hyper-parameters are tuned by grid search on the development dataset. The product and reviewer embeddings are randomly initialized from a uniform distribution (Socher et al., 2013). The word embeddings are initialized with 100-dimensions vectors pre-trained by the CBOW model (Word2Vec) (Mikolov et al., 2013).As Table 4 showed, our model observably performs better in detecting review spam for the cold-start task in both hotel and restaurant domains.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Features P R F1 A LF 54.5 71.1 61.7 55.9 1
LF+BF 63.4 52.6 57.5 61.1 2 BF EditSim+LF 55.3 69.7 61.6 56.6 3 BF W2Vsim+W2V 58.4 65.9 61.9 59.5 4 Ours RE 62.1 68.3 65.1 63.3 5 Ours RE+RRE+PRE 63.6 71.2 67.2 65.4 6 (a) Hotel
P R F1 A 53.8 80.8 64.6 55.8 1 58.1 61.2 59.6 58.5 2 53.9 82.2 65.1 56.0 3 56.3 73.4 63.7 58.2 4 58.4 75.1 65.7 60.8 5 59.0 78.8 67.5 62.0 6
(b) Restaurant
Table 4: SVM classification results across linguistic features (LF, bigrams here (Mukherjee et al., 2013b)), behavioral features (BF: RL, RD, MCS (Mukherjee et al., 2013b)); the SVM classification results by the intuitive method that finding the most similar existing review by edit distance ratio and take the found reviewers’ behavioral features as approximation (BF EditSim+LF), and results by the intuitive method that finding the most similar existing review by averaged pre-trained word embeddings (using Word2Vec) (BF W2Vsim+W2V); and the SVM classification results across the learnt review embeddings (RE), the learnt review’s rating embeddings (RRE), the learnt product’s average rating embeddings (PRE) by our model. Improvements of our model are statistically significant with p<0.005 based on paired t-test.
Review Embeddings Compared with the traditional linguistic features, e.g., bigrams, using the review embeddings learnt by our model, results in around 3.4% improvement in F1 and around 7.4% improvement in A in the hotel domain (1.1% in F1 and 5.0% in A for the restaurant domain, shown in Tabel 4 (a,b) rows 1, 5). Compared with the combination of the bigrams and the traditional behavioral features, using the review embeddings learnt by our model, results in around 7.6% improvement in F1 and around 2.2% improvement in A in the hotel domain (6.1% in F1 and 2.3% in A for the restaurant domain, shown in Tabel 4 (a,b) rows 2, 5). The F1-Score (F1) of the classification under the balance distribution reflects the ability of detecting the review spam. The accuracy (A) of the classification under the balance distribution reflects the ability of identifying both the review spam and the real review. The experiment results indicate that our model performs significantly better than the traditional methods in F1 and A at the same time. The learnt review embeddings with encoded linguistic and behavioral information are more effective in detecting review spam for the cold-start task. Rating Embeddings As we referred in Section 4.3, the rating of a review usually means the sentiment polarity of a real reviewer or the motivation of a spammer. As shown in Table 4 (a,b) rows 6, adding the rating embeddings of the products (hotel/restaurant) and reviews renders even higher F1 and A. We suppose that different rating embeddings are encoded with different semantic meanings. They reflect the semantic divergences be-
tween the average rating of the product and the review rating. In results, using RE+RRE+PRE which makes the best performance of our model, results in around 5.5% improvement in F1 and around 9.5% improvement in A in the hotel domain (2.9% in F1 and 6.2% in A for the restaurant domain, shown in Tabel 4 (a,b) rows 1, 6), compared with the LF. Using RE+RRE+PRE results in around 9.7% improvement in F1 and around 4.3% improvement in A in the hotel domain (7.9% in F1 and 3.5% in A for the restaurant domain, shown in Tabel 4 (a,b) rows 2, 6), compared with the LF+BF.
The experiment results proves that our model is effective. The improvements in both the F1 and A prove that our model performs well in both detecting the review spam and identifying the real review. Furthermore, the improvements in both the hotel and restaurant domains prove that our model possesses preferable domain-adaptability 2. It can learn to represent the reviews with global linguistic and behavioral information from large scale unlabeled existing reviews.
5.3 Our Jointly Embeddings v.s. the Intuitive Methods
As mentioned in Section 1, to approximate the behavioral information of the new reviewers, there are other intuitive methods. So we conduct experiments with two intuitive methods as com-
2The improvements in hotel domain are greater than that in restaurant domain. The possible reason is the proportion of the available training data in hotel domain is higher than that in restaurant domain (99.01% vs. 97.40% in Table 2).
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Features P R F1 A LF 54.5 71.1 61.7 55.9 1 Ours CNN 61.2 51.7 56.1 59.5 2 Ours RE 62.1 68.3 65.1 63.3 3
(a) Hotel
P R F1 A 53.8 80.8 64.6 55.8 1 56.9 58.8 57.8 57.1 2 58.4 75.1 65.7 60.8 3
(b) Restaurant
Table 5: SVM classification results across linguistic features (LF, bigrams here (Mukherjee et al., 2013b)), the learnt review embeddings (RE) ; and the classification results by only using our CNN. Both training and testing use balanced data (50:50). Improvements of our model are statistically significant with p<0.005 based on paired t-test.
parison. One is finding the most similar existing review by edit distance ratio and taking the found reviewers’ behavioral features as approximation, and then training the classifier on the behavioral features and bigrams (BF EditSim+LF). The other is finding the most similar existing review by cosine similarity of review embeddings which is the average of the pre-trained word embeddings (using Word2Vec), and then training the classifier on the behavioral features and review embeddings (BF W2Vsim+W2V). As shown in Table 4, our jointly embeddings (Ours RE and Ours RE+RRE+PRE) obviously perform better than the intuitive methods, such as the Ours RE is 3.8% (Accuracy) and 3.2% (F1) better than BF W2Vsim+W2V in the hotel domain. The experiments indicate that, our jointly embeddings do a better job in capturing the reviewer’s characteristics and modeling the correlation of textual and behavioral information.
5.4 The Effectiveness of Encoding the Global
Behavioral Information
To further evaluate the effectiveness of encoding the global behavioral information in our model, we build an independent supervised convolutional neural network which has the same structure and parameter settings with the CNN part of our model. There is not any review graphic or behavioral information in this independent supervised CNN (Tabel 5 (a,b) row 2). As shown in Tabel 5 (a,b) rows 2, 3, compared with the review embeddings learnt by the independent supervised CNN, using the review embeddings learnt by our model results in around 9.0% improvement in F1 and around 3.8% improvement in A in the hotel domain (7.9% in F1 and 3.7% in A for the restaurant domain. The results show that our model can represent the new reviews posted by the new reviewers with the correlated behavioral information encoded in the word embeddings. The transE part of our model has effectively recorded the behavioral informa-
tion of the review graph. Thus, our model is more effective by jointly embedding the textual and behavioral informations, it helps to augment the possible behavioral information of the new reviewer.
5.5 The Effectiveness of CNN
Compared with the the most effective linguistic features, e.g., bigrams, our independent supervised convolutional neural network performs better in A than F1 (shown in Tabel 4 (a,b) rows 1, 2). It indicates that the CNN do a better job in identifying the real review than the review spam. We suppose that the possible reason is that the CNN is good at modeling the different semantic aspects of a review. And the real reviewers usually tend to describe different aspects of a hotel or restaurant according to their real personal experiences, but the spammers can only forge fake reviews with their own infinite imagination. Mukherjee et al. (2013b) also proved that different psychological states of the minds of the spammers and non-spammers, lead to significant linguistic differences between review spam and non-spam.
6 Conclusion and Future Work
This paper analyzes the importance and difficulty of the cold-start challenge in review spam combat. We propose a neural network model that jointly embeds the existing textual and behavioral information for detecting review spam in the coldstart task. It can learn to represent the new review of the new reviewer with the similar textual information and the correlated behavioral information in an unsupervised way. Then, a classifier is applied to detect the review spam. Experimental results prove the proposed model achieves an effective performance and possesses preferable domain-adaptability. It is also applicable to a large scale dataset in an unsupervised way. To our best knowledge, this is the first work to handle the coldstart problem in review spam detection. We are going to explore more effective models in future.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This paper investigates the cold-start problem in review spam detection. The authors first qualitatively and quantitatively analyze the cold-start problem. They observe that there is no enough prior data from a new user in this realistic scenario. The traditional features fail to help to identify review spam. Instead, they turn to rely on the abundant textual and behavioral information of the existing reviewer to augment the information of a new user. In specific, they propose a neural network to represent the review of the new reviewer with the learnt word embedding and jointly encoded behavioral information. In the experiments, the authors make comparisons with traditional methods, and show the effectiveness of their model.  - Strengths:  The paper is well organized and clearly written. The idea of jointly encoding texts and behaviors is interesting. The cold-start problem is actually an urgent problem to several online review analysis applications. In my knowledge, the previous work has not yet attempted to tackle this problem. This paper is meaningful and presents a reasonable analysis. And the results of the proposed model can also be available for downstream detection models.  - Weaknesses:  In experiments, the author set the window width of the filters in the CNN module to 2. Did the author try other window widths, for example width `1' to extract unigram features, `3' to trigram, or use them together?  The authors may add more details about the previous work in the related work section. More specifically description would help the readers to understand the task clearly.  There are also some typos to be corrected: Sec 1: ``...making purchase decision...'' should be ``making a the purchase decision'' Sec 1: ``...are devoted to explore... '' should be `` are devoted to exploring'' Sec 1: ``...there is on sufficient behaviors...'' should be  there are no sufficient behaviors'' Sec 1: ``...on business trip...'' should be ``on a business trip'' Sec 1: ``...there are abundant behavior information...'' should be ``there is abundant behavior'' Sec 3: ``The new reviewer only provide us...'' should be ``...The new reviewer only provides us...'' Sec 3: ``...features need not to take much...'' should be ``...features need not take much...'' Sec 4: ``...there is not any historical reviews...'' should be ``...there are not any historical reviews...'' Sec 4: ``...utilizing a embedding learning model...'' should be ``...utilizing an embedding learning model...'' Sec 5.2 ``...The experiment results proves...'' should be ``...The experiment results prove...''  - General Discussion:  It is a good paper and should be accepted by ACL.


We propose a new, simple, yet effective framework, phrasal recurrent neural networks (pRNN), for language modeling and machine translation. Different from previous RNN-based language models, pRNNs store the sentential history as a set of candidate phrases with different lengths that precede the word to predict. To represent phrases as fix-length realvalued vectors, we build the RNN pyramid, which is composed of shifted parallel RNN sequences. When predicting the next word, pRNNs employ a soft attention mechanism to selective and combine the suggestions of candidate phrases. We test our model on language model and machine translation tasks. Our model leads to an improvement of over 10 points in perplexity both on standard Penn Treebank and FBIS English data set over a state-of-the-art LSTM language modeling baseline. We also apply pRNNs to encode the source sentence of machine translation besides a conventional bi-direction encoder, which improves over the Moses (phrase-based statistical model) and a strong sequence-to-sequence baseline in the Chinese-English machine translation task.
1 Introduction
There are latent nest structures beyond sequential surface words in natural language (Chomsky, 1957). In the last two decades, researchers incorporated more and more rich structural information into conventional language model (Jelinek and Lafferty, 1991; Chelba, 1997; Chelba and Jelinek, 2000; Emami and Jelinek, 2005) and
more recently on neural network-based language modeling (Dyer et al., 2016). Among the effort, one direction is to explore sub-word structures(Costa-Jussà and Fonollosa, 2016; Chung et al., 2016; Luong and Manning, 2016; Sennrich et al., 2015), such as characters, mostly to handle out-of-vocabulary word problem. A somewhat opposite direction is to explore hyper-word structure. For example, (Eriguchi et al., 2016) adopts parsing tree in encoder phase in machine translation, (Stahlberg et al., 2016) proposes to use hierarchical phrase-based (HPB) model to guide the search in decoding. Both models, however, rely heavily on human labeled data on the language structures, which is extremely expensive and limited in scale.
In this paper, we propose phrasal recurrent neural networks (pRNNs; §2), a general framework of RNNs (Elman, 1990) that explicitly models task-specific nested phrases from plain text. Here we use “phrase” as its definition in phrase-based statistical machine translation (PBSMT(Zens et al., 2002; Koehn et al., 2003)), which indicates any continues sequences of words. What different here are pRNNs permit phrases with arbitrary lengths instead of limiting them for the computational issue. The phrases in pRNNs are composed and selected in a way that is jointly learned in the language modeling, therefore requiring no human-labeled data or external model such as word alignment. In previous RNN-based language modeling, the hidden state of RNN before the word to predict summarizes the history of all previous words. Similarly, in pRNNs, we use the all state of all parallel RNNs (with the same parameters) to capture the history of all subsequence of words that precede the word to predict, with the starting word shifting from the first word the one right before the word to predict.
This set of RNNs applied parallelly to different choices of word sequences are called RNN pyra-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
mid. While most of those RNNs’ status deal with incorrect word sequences: they could either start in the middle of a chunk, or in a place too early or too late for the prediction tasks, we left it to an attention mechanism to select and combine, therefore eliminate the need for external knowledge on chunking and composition. This mechanism will be trained jointly with the composition models in pRNNs in optimizing a designed objective function, e.g, perplexity or likelihood. With proper composition function in pRNNs, the RNN pyramid provides a “phrase forest”, which could potentially contain a fairly deep nested structure in some of its members.
Our pRNN models have two merits:
• They represent all phrases in the same vector space in an explicit and unsupervised way. Which shows the potential to discover and utilize hidden structures of surface word sequences.
• They explore the possibility of network construction in another dimension: Parallel. Instead of stacking deeper and deeper layers of RNNs.
Experiments show that pRNNs are effective for language modeling (§4). Our model obtains significant better perplexities than state-of-the-art sequential Long-Short Term (LSTM) model on language modeling task, both on PTB and FBIS English data set. We also apply pRNNs to encode the source sentence of machine translation besides a conventional bi-direction encoder, which improves over the Moses (phrase-based statistical model) and a strong sequence-to-sequence baseline in the Chinese-English machine translation task.
2 Phrasal RNNs
We assume that, in the task of language model and machine translation, selecting the appropriate hidden structures for one sentence is highly related to the performance of the task.
Formally, a typical pRNN consist of three subnetworks: phrasal part P , attention part A and sequential part S. Each sub-network plays its own role and collaborates with others. P (§2.1) constructs the neural structures (realvalued vectors) which corresponding to natural linguistic structures (phrases). It takes embed-
dings (x?) of all words in a candidate phrase, as input, then output one fix-length real-valued vector p as the distributed representation (Hinton et al., 1986) of the phrase;
p = P (xji ) (1)
Where xji = [xi, · · · , xj ] (2)
A (§2.2) compares the candidate phrase regards to the current situation, give a probabilistic distribution over them, then provides the weighted sum of their representations. It takes previous calculated candidate set {p} as input, then output a weighted sum of {p} as p̂ with the help of current hidden state (ht) at time step t given by (S).
p̂ = A(h, pk, · · · , pl) (3)
S (§2.2) combine the weighted sum of candidate phrases into original RNN, forces RNN taking the structural history information into consideration. It is similar to original RNN except for one point: when predict next hidden state ht+1, besides ht and xt, it takes p̂ as input too.
ht = S(ht−1, xt, p̂t) (4)
2.1 Represent Phrases
There are many types of neural networks which can transfer phrase into distributed representation. However, when we need to handle arbitrary length phrases, another way of saying, the entire history
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
of words, the choices are very limited to a few RNNs’ variants.
Even when we choose RNNs to construct structure vectors, we are still facing a big problem. Because the hidden state ht are considered to encode the entire history information from the beginning of the sentence xt1, they can be utilized as representations of phrases begin at the sentence head {xj1|1 ≤ j ≤ t}. But they do not provide the representations of phrases which do not begin with the first word of the sentence {xji |1 < i < j ≤ t}.
To represent all candidate phrases in a sentence with n words, we build a RNN pyramid (RNNP (Fig. 3)), with n horizontal parallel RNNs {RNNn}Nn=1. RNNn indicates that it begins at the n-th word of the sentence. With all N(N + 1)/2 hidden status generated by RNN pyramid, we obtain distributed representation of all candidate phrases/structures of a sentence.
To keep consistent among these parallel RNNs in the pyramid and to limit the number of parameters for keeping the model simple, we let all parallel RNNs share the same network parameters (W,U, b).
hnt = σ(Wxt + Uh n t−1 + b) (5)
Where hnt of the RNNn indicates the hidden state of the t-th word in the sentence.
This method is kind of similar to the sharing parameters between filters of convolutional neural network (Cun et al., 1990), except for it working on the time axis, which recognizes the local invariant along each time steps.
With RNN pyramid built on a sentence, we can map all potential phrases with varying lengths into real-valued fix-length vectors. These vectors are representations of candidate structures we plan to compare at next stage.
2.2 Utilize Phrases
With the candidate structures represented by a fixlength vector (Fig. 2), we can easily apply attention mechanism on these vectors, and soft combine them to output a weighted sum as the best structure selected:
ŝt = ∑ t,n αt,nh n t−1 (6)
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Figure 3: On a 4-word sentence, RNN pyramid (dashed line triangle) generated by 4 horizontal parallel RNNs, each begins at 1 of the 4 words in the sentence. Initial status are indicated by circles. Because hidden state is considered containing all history information. The set of all hidden status in the pyramid can be mapped one-to-one to the representation of all candidate phrases of the sentence.
Where the weight of hidden state (structure) αt,n can be represented by the following form:
αt,n = exp(et,n)∑
t,n exp(et,n)
(7)
In which we define et,n as:
et,n = a(hk, ht,n) (8)
Here we combine a(hk, ht,n) with one layer of feedforward neural network, where hk is the k-th word of sequential part S.
We adopt the attention mechanism from Bahdanau et. al. (2014). As we showed in Fig 4. We put ŝ into R part of the network, let the network to combine it with h and x to predict next hidden state. We also apply our model on machine translation task within successful EncoderDecoder framework as in Fig 5.
Figure 4: Combine best structure ŝ given by attention partA in each predicting step of sequential part S. To reduce calculation, we limit the candidate phrase set at each step to the newly generated ones (the blue rectangles with solid boundaries), which means to ignore structures generated at previous steps (grey ones with dashed boundaries), therefore reduce the scale of candidate phrases set from O(N2) to O(N). The Pyramid and Seq part share the same embedding in experiments, we draw them separately in the diagram just for clearance.
3 LM Experiment
3.1 Data
To make experiment comparable with other methods, we apply our models on language model task, evaluate it in perplexity on the widely used English Penn Treebank (PTB) (Marcus et al., 1993), which pre-processing and splitting by Mikolov (2010). The data is utilized as following: sections (0-20) with 929k tokens are used for training, sections (21-22) with 73k tokens are held out as validation, and sections (23-24) with 82k tokens are used for testing. There are only top 10,000 highfrequency words are kept in the corpus. All rest low-frequency words are replaced with UNK tag. This version of data is widely used among the language modeling community. It is publicly avail-
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Figure 5: We use our pRNNs as the pyramid encoder to represent more structural information (all candidate phrases) of source sentences, just beside the original bi-direction encoder which only represents surface word sequence explictly. Then we join two context vectors from two encoders into a larger one. Then we allow the decoder to choose which portion of the larger context (which source words or candidate phrases) is more relevant to the next generated word of the target sentence. We adopt two settings of pyramid encoder, one takes only last status (the blue rectangles with solid boundaries) of each RNN as the input of attention part (src-pyr-last in Fig 4), the other takes all status (all rectangles including the grey ones with dashed boundaries) of all RNNs as the input of attention part (src-pyr-all in Fig 4).
able.1
Because the scale of PTB corpus is relatively small, we also train our model on larger FBIS English(LDC2003E14). Accordingly, we only keep top 40,000 high-frequency words in the corpus, replace rest low-frequency word with UNK tag. We use NIST MT06 as the validation set, NIST MT08 as the test set.
train valid test FBIS MT06 MT08
Sequences 219,280 6,560 5,424 Tokens 7,877,650 190,065 166,937 Types 49,210 8,476 9,576
Table 1: FBIS and NIST MT Corpus statistics.
3.2 Model Configuration
Baseline In this paper, we utilize state-of-theart LSTM framework on language model proposed by Zaremba (2014) as the baseline model. Firstly, we stack 2 layers of LSTMs, to explore more abstracted patterns which are not supposed to be discovered by a single layer. Secondly, to increase the model’s capability of noise tolerance and reduce the overfitting to training data, we introduce dropout (Hinton et al., 2012) before and after each recurrent layer. We choose dropout rate as 0.5 without tuning. For training, we used AdaDelta (Zeiler, 2012). To handle nan and inf which occasionally occur in gradient, we normalize the gradient of each batch to 1.0 and drop the parameter updates on such batches. To determine when to stop training, we set patience to 100. We set the dimension of both word embedding and hidden dimensions to 200. We initialized all parameters according to recommendations given in (Zaremba et al., 2014) and blocks(van Merriënboer et al., 2015). For all models, we used word embedding, hidden dimensions of 200 and 2-layer LSTMs. For both models, we choose dropout rate as 0.5. For training, we used AdaDelta and normalize the gradient to 1.0, set patience to 100. We initialized all parameters according to recommendations given in Zaremba et al. (2014).
Phrasal RNN We configure our model exactly as the baseline model, except adding an extra RNN
1http://www.fit.vutbr.cz/˜imikolov/ rnnlm/
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
pyramid layer above baseline’s 2-layer LSTM. We also add dropout between 2nd LSTM layer and RNN pyramid layer. We set dimension of hidden state in RNN pyramid layer as 200 either. We utilize Gated Recurrent Unit (GRU) (Chung et al., 2014) to construct RNN pyramid layer. We also tried a simplified version of GRU to build the pyramid (pRNNv in table 2), which achieves the best result.
3.3 Results
Model Perplexity 5-gram, KN5 141.2 FFNN-LM 140.2 RNN 124.7 LSTM 126 genCNN 116.4 LSTM (baseline) 106.9 pRNN 97.6 pRNNv 94.5
Table 2: Perplexity on PENN TREEBANK, where the top 5 rows of numbers are results reported in previous work, our baseline and new pRNN model are in last three rows.
Model Perplexity 5-gram, KN5 278.6 FFNN-LM(5-gram) 248.3 FFNN-LM(20-gram) 228.2 RNN 223.4 LSTM 206.9 genCNN 181.2 LSTM (baseline) 171.8 pRNN 161.5
Table 3: Perplexity on FBIS data set, where the top 5 rows of numbers are results reported in previous work, our baseline and new pRNN model are in last two rows.
We report our perplexities result of language model in table (2 and 3). We calculate perplexity over a sequence [w1, . . . , wn] with
PPL = exp ( − log(Prob(w n 1 ))
n
) (9)
(including the end of sentence (EOS) symbol). pRNN and its variant outperform over 10 points of
ppl over a strong baseline on both PTB and FBIS English data set.
4 MT Experiment
4.1 Data
We evaluate all three models, PBSMT, RNNsearch, pRNN on the same data set. We utilize 1.25M sentence pairs, which are extracted from LDC corpora as training data. There are 34.5 English words and 27.9M Chinese words in the training data. We select NIST 2002 (MT02) data set as our development set, the NIST 2003 (MT03), NIST 2004 (MT04), NIST 2005 (MT05), NIST 2006 (MT06) and NIST 2008 (MT08) as test sets. When training neural networks, we limit the size of vocabularies of both source and target side to the most frequent 16K words. All rest low-frequency words are replaced with UNK tag. Chinese vocabulary covers approximately 95.8% of the corpora. English vocabulary covers approximately 98.3% of the corpora.
4.2 Model Configuration
Baseline In this paper, we use an open-source implementation (Meng et al., 2015) of RNNsearch (Bahdanau et al., 2014) as baseline model. To increase the model’s capability of noise tolerance and reduce the overfitting to training data, we introduce dropout (Hinton et al., 2012) before softmax layer, and set the dropout rate equal to 0.5. We choose dropout rate as 0.5 without tuning. For training, we used AdaDelta (Zeiler, 2012) and normalize the gradient to 1.0, . To handle nan and inf which occasionally occur in gradient, we normalize the gradient of each batch to 1.0 and drop the parameter updates on such batches. We set the dimension of both source and target word embedding as 620, and hidden dimensions to 1000. We initialized all parameters according to recommendations given in (Bahdanau et al., 2014). We also introduce the phrase-based model of Moses (Koehn et al., 2007) as a secondary baseline too.
Phrase-based NMT We configure our model exactly as the baseline model, except adding an extra RNN pyramid as a secondary source encoder. As the limitation on the memory of GPU, we keep only phrases ended at eos into consideration. Thus we name it src-pyr-last in table 4. We set the dimension of hidden dimensions inside pyramid as 1000 too. We utilize Gated Recur-
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Models MT02 MT03 MT04 MT05 MT06 MT08 Test Avg. Diff moses 33.41 31.61 33.48 30.75 31.07 23.37 30.056 +0.754 RNNsearch (groundhog) 32.32 29.02 31.25 28.32 27.99 20.29 27.374 -1.928 RNNsearch (baseline) 34.28 30.61 33.24 30.66 29.83 22.17 29.302 +0.000 pRNN (src-pyr-last) 35.48 31.61 34.40 31.96 31.36 22.82 30.430 +1.128 pRNN (src-pyr-all) 35.49 32.08 34.51 31.81 30.91 22.86 30.434 +1.132
Table 4: BLEU score on 1.25M training corpus with 16k dictionary on both source and target side. The above two lines of the table are results of open-source machine translation systems. Bold numbers indicate the best results on the data set (column). pRNNs are better than original RNNsearch model baseline (in-house reimplemented). We find it is interesting that results of src-pyr-all are only slightly better than src-pyr-last, we guess this is due to the limited discriminative power of simple attention mechanism when meeting large number of complex candidates.
rent Unit (GRU) (Chung et al., 2014) to construct RNN pyramid layer.
For a fair comparison, we run baseline system (RNNsearch) many times (not epoch) and report only the best one. We only run PBNMT once.
4.3 Results
We report our BLEU result of three models in table (4). We use the case-insensitive 4-gram NIST BLEU (Papineni et al., 2002) score given by mteval v11.pl pRNNs outperforms both PBSMT and Encoder-Decoder model.
5 Related Work
5.1 Relation to Previous Attempts on
Structural Information
In the last two decades, to achieve better performance, researchers incorporated more and more rich structural information into conventional model (Jelinek and Lafferty, 1991; Chelba, 1997; Chelba and Jelinek, 2000; Emami and Jelinek, 2005). This trend is more clear in statistical machine translation (SMT) community, from wordbased SMT (Brown et al., 1993) to phrase-based SMT (Koehn et al., 2003), hierarchical phrasebased SMT (Chiang, 2005) and syntax-based SMT (Huang et al., 2006; Liu et al., 2006; Galley et al., 2006; Xie et al., 2011) (forest-based (Mi and Huang, 2008; Mi et al., 2008)). among them, phrase-based SMT (Zens et al., 2002; Koehn et al., 2003) was the most widely adopted translation model. The Same trend can be observed in neural network strand. There were many works which successfully modeled structure in neural network on parsing (Dyer et al., 2016; Emami and Jelinek, 2005; Henderson, 2004; Titov and Hender-
son, 2007; Buys and Blunsom, 2015) or language modeling tasks (Dyer et al., 2016; Chelba and Jelinek, 2000; Emami and Jelinek, 2005; Chelba, 1997).
In neural machine translation (NMT) area, the situation is more complex. In recent years, the most popular and success NMT model is EncoderDecoder model (Bahdanau et al., 2014; Sutskever et al., 2014). Which has achieved competitive or better results in many translation tasks(Luong et al., 2015a,b). However, beyond sequential surface words, there are latent nest structures in natural language (Chomsky, 1957). One direction to explore is to introduce sub-word structures(CostaJussà and Fonollosa, 2016; Chung et al., 2016; Luong and Manning, 2016; Sennrich et al., 2015), such as characters. The most important reason to dig into sub-word is to handle out-of-vocabulary word problem. This problem is rooted in the limited size of vocabulary, which utilized by NMT mapping symbols to real-valued dense vector.
Another direction is to explore hyper-word structure. Authors of (Eriguchi et al., 2016) adopted parsing tree in encoder phase. However, this method depends heavily on human-labeled data, which is always expensive and limited in scale. Authors of (Stahlberg et al., 2016) introduce hierarchical phrase-based (HPB) model as the guider of search space. However, the HPB model and NMT model are trained separately and combined only when decoding. Compare this to the previous method, (Eriguchi et al., 2016) can be categorized into introducing external data, (Stahlberg et al., 2016) can be categorized into introducing external model. In an ideal situation, all external model can be replaced by a neural net-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
work with equal ability.
5.2 Similarity to Deep Memory Network
Deep Memory Network is an effective implementation of the neural turing machine. When they use state machine such as GRU or LSTM to read from one memory and write to another, the memory IO addresses are either content-based (via attention mechanism) or location-based (actually sequentially cell-by-cell) (Meng et al., 2015). However, there are much more other methods in the location-based category.
Cell-by-cell vs. Incremental If we consider the hidden of one time step inside pyramid RNN as memory, we can name the operations as incremental read and write. The intuition behind incremental addressing is, when we read little, we know little, we only have the ability to write little. But when we read more, we know more, we are gone to have the ability to write more.
6 Discussion
Our experiments clearly show that the proposed pRNN model is quite effective in language modeling and machine translation. This is the because of:
• when model predicting, it is provided with all candidate phrases as structure information rather than just surface sequential words.
• utilizing attention mechanism to compare and combine to get the weighted sum which best fit for predicting next hidden state.
The most significant question that remains is how well the quality of forest generated as a by-product of pRNN, will it get a better result than other supervised parsing model trained on human label data.
7 Conclusion
We introduced phrasal recurrent neural network, an RNN model with all potential candidate phrases considered. Our model does not require any human labeled data to construct the structures. It outperforms the state-of-the-art LSTM language models. Our model does not require any external resources such as human labeled data or word align model to construct the phrases. It outperforms both state-of-the-art PBSMT and RNNsearch model.
We make two main contributions:
• Instead of packing all information in distributed representation and internal hidden status, which are computing-friendly, we try to represent natural structure in an explicit way, which are human-friendly.
• Instead of stacking deeper and deeper layers of RNNs, we explore the possibility of network construction in another dimension: making RNN sequences parallel.

This paper proposed a new phrasal RNN architecture for sequence to sequence generation. They have evaluated their architecture based on (i) the language modelling test evaluated on PTB and FBIS and (ii) Chinese-English machine translation task on NIST MT02-08 evaluation sets. The phrasal RNN (pRNN) architecture is achieved by generating subnetworks of phrases.   Strengths ====  A new phrasal architecture.   Weaknesses ====  **Technical**:   It's unclear whether there is a limit set  on the phrase length of the pRNN. Maybe I've missed this in the paper, if there is, please be more explicit about it because it affects the model quite drastically if for every sentence the largest phrase length is the sentence length.    - It's because if the largest phrase length is the sentence length, then model can be simplified into a some sort of convolution RNN where the each state of the RNN goes through some convolution layer before a final softmax and attention.    - If there is a limit set on the phrase length of pRNN, then it makes the system more tractable. But that would also mean that the phrases are determined by token ngrams which produces a sliding window of the  pyramid encoders  for each sentence where there are instance where the parameter for these phrases will be set close to zero to disable the phrases and these phrases would be a good intrinsic evaluation of the pRNN in addition to evaluating it purely on perplexity and BLEU extrinsically.   The usage of attention mechanism without some sort of pruning might be problematic at the phrasal level. The author have opted for some sort of greedy pruning as described in the caption of figure 4. But I support given a fixed set of phrase pairs at train time, the attention mechanism at the phrasal level can be pre-computed but at inference (apply the attention on new data at test time), this might be kind of problematic when the architecture is scaled to a larger dataset.   **Empirical**:   One issue with the language modelling experiment is the choice of evaluation and train set. Possibly a dataset like common crawl or enwiki8 would be more appropriate for language modelling experiments.   The main issue of the paper is in the experiments and results reporting, it needs quite a bit of reworking.    - The evaluation on PTB (table 2) isn't a fair one since the model was trained on a larger corpus (FBIS) and then tested on PTB. The fact that the previous study reported a 126 perplexity baseline using LSTM and the LSTM's perplexity of 106.9 provided by the author showed that the FBIS gives an advantage to computing the language model's perplexity when tested on PTB.   - Also, regarding section 3.3, please cite appropriate publications the  previous work  presented in the tables. And are the previous work using the same training set?   - Additionally, why isn't the the GRU version of pRNNv reported in the FBIS evaluation in Table 3?  The result section cannot be simply presenting a table without explanation:   - Still on the result sections, although it's clear that BLEU and perplexity are objective automatic measure to evaluate the new architecture. It's not really okay to put up the tables and show the perplexity and BLEU scores without some explanation. E.g. in Table 2, it's necessary to explain why the LSTM's perplexity from previous work is higher than the author's implementation. Same in Table 3.   The result presented in Table 4 don't match the description in Section 4.3:   - It's not true that the pRNN outperforms both PBSMT and Enc-Dec model. The authors should make it clear that on different evaluation sets, the scores differs. And it's the averaged test scores that pRNN performs better  - Please also make it clear whether the  Test Avg.  is a micro-average (all testsets are concatenated and evaluated as one set) or macro-average (average taken across the scores of individual test sets) score.   For table 4, please also include the significance of the BLEU improvement made by the pRNN with respect to the the baseline, see https:  github.com jhclark multeval  General Discussion ====  As the main contribution of this work is on the phrasal effect of the new RNN architecture, it's rather important to show that the phrases are more coherent than the vanilla LSTM   RNN model. Thus the BLEU evaluation is insufficient. A closer look at evaluating the phrases in a subset of the evaluation set would be necessary to support the claims.   Does the baseline system (groundhog) contains the attention mechanism?    - If so, please be more specific in describing it in section 4.2 and Table 4.    - If not, please remove the attention layer after the encoder in figure 5. Also, the lack of attention mechanism provides a disadvantage to the baseline enc-dec system and it's unclear whether the pRNN can outperform or be an additive feature to the enc-dec system with an attention mechanism. The unfair disadvantage is even more prevalent when the pRNN uses multiple phrasal attention layers within a single sentence while a simple enc-dec system without attention is used as a benchmark =(  Question: Wouldn't a simpler way to get phrasal RNN is to put the  pyramid  RNNs of a phrase into some soft of a average pooling layer?  Minor Issues  ====  Figure 2 is a little redundant, I think figure 1 is enough to compare it against the pRNN (figure3 and 4).  Also, possibly figure 3 can be combined into the pyramid part of figure 4. And more space can be freed up to further explain the results section.   Please don't abuse figure table captions, whenever possible, please try to keep the description of the tables and figures in-text.    **Please put the verbose caption description in the main text for Figure 3, 4, 5 and Table 4**  Spacing in between some of the equations can also be reduced (e.g. in latex use )