A New Formula for Vietnamese Text Readability Assessment
Text readability has an important role in text drafting and document selecting. Researches on the readability of the text have been made long ago for English and some common languages. There are few researches in Vietnamese text readability and most of them are performed from more than two decades ago on very small corpora. In this paper, we build a new and larger corpus and use it to create a newer formula to predict the difficulty of Vietnamese text. The experimental results show that the new formula can predict the readability of Vietnamese documents with over 80% accuracy.
1 Introduction
Text readability – as defined by Edgar Dale and Jeanne Chall (Dale and Chall, 1949) – is “the sum total (including all the interactions) of all those elements within a given piece of printed material that affect the success a group of readers has with it. The success is the extent to which they understand it, read it at an optimal speed, and find it interesting.” Text readability has a huge impact on the reading and comprehending a text. Base on the readability, readers can determine whether a text is suitable for their reading ability or not. The text author(s) can also use the readability of the draft to guide readers object or have some adjustments to make it fit the toward reader. Building a model to analyze text readability has meant a lot in the scientific and practical: help scientists writing research reports more readable; support educators drafting textbooks and curricula to suit each age of students; support publishers in shaping the audience; help governments drafting legal documents to suit the majority of citi-
zens; or to assist manufacturers in preparing user guide for their products. . . In addition, text readability can effectively support in choosing appropriate curriculums when teaching language for foreigners.
Researches on text readability have begun since the early of the 20th century, most of them are for English and some common languages. Most famous studies in text readability are creating linear functions to assess and grade documents like Dale-Chall formula (Dale and Chall, 1949), Flesch Reading Ease formula (Flesch, 1949), FleschKincaid formula (Kincaid et al., 1975), Gunning Fog formula (Robert, 1952), SMOG formula (Laughlin, 1969). . . In Vietnamese, there are only two studies on text readability of Liem Thanh Nguyen and Alan B. Henkin in 1982 and 1985. Both these two researches focus on examining relations between statistical characteristics at words level and at sentences level and text readability. However, these two works only evaluate on small corpora of 20 documents (Nguyen and Henkin, 1982) and 54 documents (Nguyen and Henkin, 1985). Since these studies, there is almost no other publication on Vietnamese text readability.
In this paper, we mainly focus on creating a new formula for Vietnamese text readability assessment base on a self-built corpus with a large number of documents. Following this, the remaining of this paper is presented as below: we first present some famous formulas in English text readability as related works in section 2; we then present our works on building a corpus and creating formulas to assess Vietnamese text readability along with some experiments and results in section 3; finally, section 4 presents our discussions and conclusions.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
2 Related works
Studies on text readability have begun since the early of the 20th century. Up to now, there are thousands of works in this field. In this section, we will describe some famous formulas for English text readability assessment and two formulas for Vietnamese. First is the new Dale-Chall formula (Chall and Dale, 1995). This is a very famous readability formula that provides the comprehension difficulty score of an English document. This uses a list of 3000 words that students in fourth-grade could understand. Words that not in this list are considered as difficult words. The score is calculated as Equation 1:
RS = 0.1579 ∗ (PDW ) + 0.0496 ∗ASL (1)
where RS is the Reading Grade of a reader who can comprehend the text at 3rd grade or below, the higher RS, the more difficult the text; PDW is Percentage of Difficult Words; and ASL is Average Sentence Length in words. The list of 3.000 words that student in fourth-grade could understand was replaced by the list of 3.000 frequent Vietnamese words to experiment in Vietnamese corpus. The second formula is Flesh Reading Ease (Flesch, 1949). This is an easy formula for measuring English text readability and is used by many US government agencies like US Department of Defense. . . It is also integrated into the Microsoft Word - the most popular word processor - from version 2007 to help users checking and controlling the readability of the document. The formula is defined as Equation 2:
RE = 206.835−(1.015 ∗ASL)−(84.6 ∗ASW ) (2) whereRE is the Readability Ease of the text;ASL is Average Sentence Length in words; and ASW is the average number of syllables per word. Like the name of the formula, the higherRE, the easier the document. Next is the Flesch-Kincaid grade level readability formula (Kincaid et al., 1975) for English. This formula is best suited for education and is also integrated into the Microsoft Word. It is defined as Equation 3:
FKRA = (0.39xASL)+(11.8 ∗ASW )−15.59 (3)
where FKRA is the Flesch-Kincaid Reading Age, indicates the grade-school level that student can read and comprehend; ASL is Average Sentence Length in words; andASW is the average number of syllables per word. Because this formula determines the grade-school level of the text so the higher FKRA, the more difficult the text. Continue is the Gunning Fog index formula (Robert, 1952). This formula is developed by Robert Gunning – an American textbook publisher. It is defined as Equation 4:
Grade level = 0.4 (ASL+ PHW ) (4)
where ASL is Average Sentence Length in words; and PHW is Percentage of Difficult Words - the percentage of words which have three or more syllables that are not proper nouns, combinations of easy words or hyphenated words, or two-syllables verbs made into three with -es and -ed endings. Similar to the Flesch-Kincaid grade level, the higher Gunning Fog index, the more difficult the text. Next is the SMOG formula (Laughlin, 1969). This formula estimates the years of education a person needs to understand a specific text and is widely used in checking health messages. The higher SMOG value, the more difficult the text. The formula is:
SMOG grade = 3 + √ Polysyllable Count (5)
In Vietnamese, as mentioned in Section 1, there are only two researches on text readability of Liem Thanh Nguyen and Alan B. Henkin in 1982 and 1985 (Nguyen and Henkin, 1982, 1985). The first formula is defined as Equation 6
RL = 2WL+ 0.2SL− 6 (6)
where RL is Readability Level of the text; WL is Average Word Length in characters; and SL is Average Sentence Length in words. The second formula was revised from the first one with the additional role of the ratio of difficult words in the document:
RL = 0.27WD + 0.13SL+ 1.74 (7)
where RL is Readability Level of the text, the higher RL value, the more difficult the text; WD is Word Difficulty; and SL is Average Sentence Length in words.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
However, these two works only evaluate on small corpora of 20 documents (Nguyen and Henkin, 1982) and 54 documents (Nguyen and Henkin, 1985). Since these studies, there is almost no other publication on Vietnamese text readability. Furthermore, as proposed by Professor Lucius Adelno Sherman, languages are still changing over time (Sherman, 1893), so researches on readability will still continue.
3 Method
3.1 Building corpus
As mentioned in sections 1 and 2, there are only two publications on Vietnamese text readability and both of them use only small corpora with 20 and 54 documents (Nguyen and Henkin, 1982, 1985). So in our research, we built another corpus with a larger amount of document for examining. 1,000 documents were collected into 3 categories of difficulty from various sources with the following criteria:
1. Easy documents: including documents written for children or by children or just need people who are studying at primary schools or having maximum primary education to read and understand. These documents were mainly collected from primary school textbooks, primary sample essays, fairy-tales, stories for babies. . .
2. Normal documents: they are documents written for middle and high school students, or documents which only need people with high school education to be readable and understandable. Most documents in this category were collected from textbooks and general newspapers.
3. Difficult documents: including documents written for college students, specialized documents, scientific paper. . . which need high or specialized education to be readable and understandable. These documents were collected from university textbooks, specialized documents, political theory articles, language and literary articles, law and legal documents. . .
Ten experts were asked to evaluate collected documents. They are Vietnamese language specialists, current or former Vietnamese literature teacher - who has much knowledge and experiment in using and teaching Vietnamese. Each document was ensured to be evaluated by 3 experts with the following instruction: each person carefully reads each given document and gives a score for that document: a) If that document is easy enough for children and people with just primary school education to read and understand, give the Score = 1; b) Score = 2 if almost all adults and people with average education can read and understand; c) If documents need people with high or specialized education to be readable and understandable, give the Score = 3. The overall scores of each document are used to re-category that document:
1. If the document got 3 equal scores: that document will be categorized according to the category of the score.
2. If the document got 2 equal scores and one different: that document will be put to the category of the 2 equal scores.
3. If the document got 3 different scores: that document will be filtered out of the corpus.
Easy Normal Difficult Overall No. documents 235 413 348 996 No. sentences 4,006 13,772 24,646 42,424 Average sentence length in word 13.59 19.24 22.71 19.12 Average sentence length in syllable 16.48 25.04 33.66 26.03 Average sentence length in character 79.50 124.39 173.54 130.97 Average word length in syllable 1.21 1.30 1.48 1.34 Average word length in character 5.82 6.44 7.62 6.71
Table 1: Statistics numbers of built corpus.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Finally, for some statistic later, we assign the readability value of 1 for all documents in the Easy category, 2 for the Normal and 3 for the Difficult category. Table 1 presents some statistic number of the final corpus. To ensure that our corpus is reliable, we calculated the Fleiss’ kappa score (Fleiss, 1971). This is a statistical measure for assessing the agreement between a fixed number of raters when classifying items. The Fleiss’ kappa measure is usually used in natural language processing to assess the reliability of agreement between references in the corpus or between manual annotation by raters/experts. The K value is 0.422, which demonstrates that there is a moderate agreement between annotators so the corpus is reliable.
3.2 Features
In this part, we will describe some features that are commonly used in text readability assessment. Average sentence length: the average sentence length of a text is one of the simplest and common characteristic when measuring text readability. In this paper, we examined three types of average sentence length as Equation 8, 9 and 10: Average Sentence Length in Words (ASLW):
ASLW = word count
sentence count (8)
Average Sentence Length in Syllables (ASLS):
ASLS = syllable count sentence count
(9)
Average Sentence Length in Characters (ASLC):
ASLC = character count sentence count
(10)
Average word length: two types of average word length were examined in this study as Equation 11 and 12: Average Word Length in Syllables (AWLS):
AWLS = syllable count word count
(11)
Average Word Length in Characters (AWLC):
AWLC = character count word count
(12)
In Vietnamese, word maybe monosyllabic or polysyllabic (compound word) with a whitespace
between each syllable; each syllable is a combination of letters with or without tonal and word marks. For example: the word “nghe” (listen) is a monosyllable with four letters (n, g, h, and e); the word “chạy” (run) is a monosyllable with four letters (c, h, a, y) and a tonal mark (.); the word “thời gian” (time) is a polysyllable (2 syllables “thời” and “gian”) with eight letters (t, h, o, i, g, i, a, n), a word mark ( ’ ), a tonal mark ( ` ) and a whitespace.
Percentage of difficult words: in many studies, the percentage of difficult words is an important feature when evaluating text readability. However, create the easy or difficult word list needs a lot of effort, so most researches used frequent word list as a replacement: if a word does not appear in the frequent list, it will be considered as a difficult word. In this study, we used two frequent lists: the first is top 1,000 frequent words extracted from the frequent word list of Dien and Hao (Dinh and Hao, 2015); and the second is 1,000 frequent words extracted from all easy documents of our built corpus. The percentage of difficult words is calculated as Equation 13:
PDWi = difficult word count
word count (13)
In this paper, PDW1 stands for a percentage of difficult words calculated using Dien and Hao’s list and PDW2 stands for a percentage of difficult words calculated using our easy list. Not only the percentage of difficult words, but also the percentage of difficult syllables was examined in our study. We also used two frequent lists: the top 1,000 frequent syllables extracted from the list of Dien and Hao (Dinh and Hao, 2015); and the 1,000 frequent syllables extracted from all easy documents of our corpus. The percentage of difficult syllables is calculated as Equation 14:
PDWi = difficult syllable count
syllable count (14)
Similar to word, in this paper, PDS1 and PDS2 stand for a percentage of difficult words calculated using Dien and Hao’s list and our easy list respectively.
3.3 Create formula
The first thing is finding which features are suitable for predicting text readability through corre-
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
TR ASLW ASLS ASLC AWLS AWLC PDS1 PDW1 PDS2 PDW2 TR 1 ASLW 0.567 1 ASLS 0.675 0.970 1 ASLC 0.695 0.955 0.997 1 AWLS 0.770 0.483 0.669 0.699 1 AWLC 0.774 0.497 0.674 0.716 0.979 1 PDS1 0.051 -0.116 -0.140 -0.132 -0.107 -0.073 1 PDW1 0.103 -0.069 -0.029 -0.018 0.156 0.156 0.680 1 PDS2 0.466 0.201 0.237 0.251 0.328 0.355 0.666 0.602 1 PDW2 0.786 0.438 0.562 0.589 0.785 0.793 0.340 0.467 0.782 1
Table 2: Correlation between each feature with the text readability and between features together (TR is the Text Readability).
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Average Coefficient values
Const -0.8141 -0.6767 -0.6372 -0.7777 -0.7419 -0.7295 ASLC 0.0037 0.0040 0.0043 0.0040 0.0040 0.0040 AWLC 0.2105 0.1779 0.1699 0.2032 0.1911 0.1905 PDW2 2.6960 2.7974 2.7224 2.6370 2.7208 2.7147
Accuracy Easy 0.7234 0.7021 0.8511 0.7234 0.8085 0.7660 Normal 0.8313 0.9036 0.8537 0.8537 0.8313 0.8571 Difficult 0.7681 0.7857 0.8000 0.8000 0.7826 0.7845 Overall 0.7839 0.8150 0.8342 0.8040 0.8090 0.8102
Table 3: Coefficient values and accuracy of all formulas.
lation analysis. Table 2 shows the correlation coefficients between each feature with the text readability and between features together performed on our corpus.
We can see that the ASLS, ASLC, AWLS, AWLC and PDW2 are high correlated with the text readability. However, they are also high correlated with some others. To choose which features to put in the formula, we select sequentially features from the highest correlation with text readability to the lowest and remove features have high correlation coefficient with selected features. Through some experiments, we decided to use the threshold 0.9 for removing features: if a feature has the correlation coefficient with selected features greater than or equal to 0.9, that feature will not be cho-
sen. Following this, three features were selected: ASLC, AWLC, and PDW2. The features AWLS and ASLS are high correlated with AWLC and ASLC so they were not chosen. The selected features were used as predictors to perform multiple regression analysis with Text Readability as the criterion. The purpose is to find coefficient values of these features to form a formula for predicting text readability. The general formula is:
Readability = A1ASLC +A2AWLC
+A3PDW2 +A4 (15)
WithA1 toA4 are coefficient values. In this study, we divided our corpus into five equal parts for cross-fold analyzing. In each fold, four part were
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
selected to perform regression analysis and the coefficient values of each fold will be used to predict the readability of the remaining part using Formula 15. The predicted values were rounded to the nearest unit and compared to the expert evaluated readability for accuracy assessment. Finally, the average coefficient value of each predictor was used to form the final text readability formula. The final formula is:
Readability = 0.004ASLC + 0.1905AWLC
+2.7147PDW2− 0.7295 (16)
Table 3 presents coefficient values and accuracy of all formulas.
4 Discussion and conclusion
From the Table 2, we can see that word length, sentence length and the percentage of difficult words still play an important role in measuring the text readability. The Percentage of difficult syllables (PDS1) and words (PDW1) calculated on the lists extracted from Dien and Hao’s lists (Dinh and Hao, 2015) are low correlated with the Text readability of the built corpus. The main reason is the lists of Dien and Hao were statistically analyzed from the corpus mainly collected from newspapers, which are mostly texts with normal readability. So the Dien and Hao’s lists and other frequent lists which were collected from normal texts may not be good replacements for the easy word and syllable list. In the Table 3, we can see that the final text readability formula can accurately predict more than 76% of the easy texts, more than 85% of the normal documents and more than 78% of the difficult texts. Overall, the formula can predict the Vietnamese text readability with the accuracy up to 81%. This is a good result and can be applied in practice. In this paper, we have presented our work on creating a new large corpus for Vietnamese text readability assessing. We also used the corpus to create a new formula for predicting Vietnamese text readability. Experiments performed on the corpus using created formula shows that the formula can predict the readability of Vietnamese text with high accuracy. For the future works, other corpora will be built with more detailed levels of difficulty and
for more specific domains. Other deeper features like part-of-speech, sentence structure, discourse. . . will be examined to create more precise formula(s). Some machine learning methods will be examined to create some classifier for automatically Vietnamese text readability assessment.

The authors present a new formula for assessing readability of Vietnamese texts. The formula is developed based on a multiple regression analysis with three features. Furthermore, the authors have developed and annotated a new text corpus with three readability classes (easy, middle, hard).  Research on languages other than English is interesting and important, especially when it comes to low-resource languages. Therefore, the corpus might be a nice additional resource for research (but it seems that the authors will not publish it - is that right?). However, I don't think the paper is convincing in its current shape or will influence future research. Here are my reasons:  - The authors provide no reasons why there is a need for delevoping a new formula for readability assessments, given that there already exist two formulas for Vietnamese with almost the same features. What are the disadvantages of those formulas and why is the new formula presented in this paper better?  - In general, the experimental section lacks comparisons with previous work and analysis of results. The authors claim that the accuracy of their formula (81% on their corpus) is  good and can be applied in practice . What would be the accuracy of other formulas that already exist and what are the pros and cons of those existing formulas compared to the new one?  - As mentioned before, an analysis of results is missing, e.g. which word   sentence lengths   number of difficult words are considered as easy middle hard by their model?  - A few examples how their formula could be applied in a practical application would be nice as well.  - The related work section is rather a  background  section since it only presents previously published formulas. What I'm missing is a more general discussion of related work. There are some papers that might be interesting for that, e.g., DuBay 2004:  The principles of readability , or Rabin 1988:  Determining difficulty levels of text written in languages other than English   - Since Vietnamese is syllable-based and not word-based, I'm wondering how the authors get  words  in their study. Do they use a particular approach for merging syllables? And if yes, which approach do they use and what's the accuracy of the approach?  - All in all, the content of the paper (experiments, comparisons, analysis, discussion, related work) is not enough for a long paper.  Additional remarks:  - The language needs improvements  - Equations: The usage of parentheses and multiplying operators is inconsistent  - Related works section: The usage of capitalized first letters is inconsistent


A Teacher-Student Framework for Zero-Resource Neural Machine Translation
While end-to-end neural machine translation (NMT) has made remarkable progress recently, it still suffers from the data scarcity problem for low resource language pairs and domains. In this paper, we propose a method for zero-resource NMT by assuming that parallel sentences have close probabilities of generating a sentence in a third language. Based on the assumption, we are able to train a source-to-target NMT model without parallel corpora available (“student”) guided by an existing pivot-to-target NMT model (“teacher”) on a source-pivot parallel corpus. Experimental results show that the proposed method significantly improves over a baseline pivot-based model by +3.0 BLEU points across various language pairs.
1 Introduction
Neural machine translation (NMT) (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2014), which directly models the translation process in an end-to-end way, has attracted intensive attention from the community. Although NMT has achieved state-of-theart translation performance on resource-rich language pairs such as English-French and GermanEnglish (Luong et al., 2015; Jean et al., 2015; Wu et al., 2016), it still suffers from the unavailability of large-scale parallel corpora for translating low resource languages. Due to the large parameter space, neural models usually learn poorly from low-count events, resulting in a poor choice for low resource language pairs. Zoph et al. (2016) indicate that NMT obtains much worse translation quality than a statistical machine translation
(SMT) system on low-resource languages. As a result, a number of authors have endeavored to explore methods for translating language pairs without parallel corpora available. These methods can be roughly divided into two broad categories: multilingual and pivot-based. Firat et al. (2016b) present a multi-way, multilingual model with shared attention to achieve zeroresource translation. They fine-tune the attention part using pseudo bilingual sentences for the zeroresource language pair. Another direction is to develop a universal NMT model in multilingual scenarios (Johnson et al., 2016; Ha et al., 2016). They use parallel corpora of multiple languages to train one single model, which is then able to translate a language pair without parallel corpora available. Although these approaches prove to be effective, the combination of multiple languages in modeling and training leads to significantly increased complexity.
Another direction is to achieve source-to-target NMT without parallel data via a pivot, which is either text (Cheng et al., 2016) or image (Nakayama and Nishida, 2016b). Cheng et al. (2016) propose a pivot-based method to zero-resource NMT: it first translates the source language to a pivot language, which is then translated to the target language. Nakayama and Nishida (2016b) show that using multimedia information as pivot also significantly benefit zero-resource translation. However, pivot-based approaches usually need to divide the decoding process into two steps, which is not only more computationally expensive, but also potentially suffers from the error propagation problem (Zhu et al., 2013).
In this paper, we propose a new method for zero-resource neural machine translation. Our method assumes that parallel sentences should have close probabilities of generating a sentence in a third language. To train a source-to-target NMT
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
(a) (b)
X YZ Z
X
Y
P (z|x;✓x!z) P (y|z;✓z!y)
P (y|z;✓z!y)
P (y|x;✓x!y)
Figure 1: (a) The pivot-based approach and (b) the teacher-student approach to zero-resource neural machine translation. X, Y, and Z denote source, target, and pivot languages, respectively. We use a dashed line to denote that there is a parallel corpus available for the connected language pair. Solid lines with arrows represent translation directions. The pivot-based approach leverages a pivot to achieve indirect source-to-target translation: it first translates x into z, which is then translated into y. Our approach directly trains the intended source-to-target model (“student”) on a source-pivot parallel corpus, with the guidance of an existing pivot-to-target model (“teacher”). Our training algorithm is based on the translation equivalence assumption: if x is a translation of z, then P (y|x;θx→y) should be close to P (y|z;θz→y).
model without parallel corpora (“student”), we leverage an existing pivot-to-target NMT model (“teacher”) to guide the learning process of the student model on a source-pivot parallel corpus. As compared with the pivot-based approaches (Cheng et al., 2016; Nakayama and Nishida, 2016b), our method allows direct modeling of the intended NMT model, without the need to divide training or decoding into two steps. This strategy not only improves efficiency but also avoids error propagation in decoding. Experiments on the Europarl and WMT datasets show that our approach achieves significant improvements in terms of both translation quality and decoding efficiency over a baseline pivot-based approach to zeroresource NMT on Spanish-French and GermanFrench translation tasks.
2 Background
Neural machine translation (Sutskever et al., 2014; Bahdanau et al., 2014) advocates the use of neural networks to model the translation process in an end-to-end manner. As a data-driven approach, NMT treats parallel corpora as the major source for acquiring translation knowledge.
Let x be a source-language sentence and y be a target-language sentence. We use P (y|x;θx→y) to denote a source-to-target neural translation model, where θx→y is a set of model parame-
ters. Given a source-target parallel corpus Dx,y, which is a set of parallel source-target sentences, the model parameters can be learned by maximizing the log-likelihood of the parallel corpus:
θ̂x→y = argmax θx→y { ∑ 〈x,y〉∈Dx,y logP (y|x;θx→y) }
Given learned model parameters θ̂x→y, the decision rule for finding the translation with the highest probability for a source sentence x is given by
ŷ = argmax y
{ P (y|x; θ̂x→y) } (1)
As a data-driven approach, NMT heavily relies on the availability of large-scale parallel corpora to deliver state-of-the-art translation performance (Wu et al., 2016). Zoph et al. (2016) report that NMT obtains much lower BLEU scores than SMT if only small-scale parallel corpora are available. The heavy dependence on the quantity of training data poses a severe challenge for NMT to translation zero-resource language pairs.
Simple and easy-to-implement, pivot-based methods have been widely used in SMT for translating zero-resource language pairs (De Gispert and Marino, 2006; Cohn and Lapata, 2007; Utiyama and Isahara, 2007; Wu and Wang, 2007;
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Bertoldi et al., 2008; Wu and Wang, 2009; Zahabi et al., 2013; Kholy et al., 2013). As pivotbased methods are agnostic to model structures, they have been adapted to NMT recently (Cheng et al., 2016; Nakayama and Nishida, 2016b).
Figure 1 illustrates the basic idea of pivot-based approaches to zero-resource NMT (Cheng et al., 2016; Nakayama and Nishida, 2016b). Let X, Y, and Z to denote source, target, and pivot languages. We use dashed lines to denote language pairs with parallel corpora available and solid lines with arrows to denote translation directions.
Intuitively, the source-to-target translation can be indirectly modeled by bridging two NMT models via a pivot:
P (y|x;θx→z,θz→y) = ∑ z P (z|x;θx→z)P (y|z;θz→y) (2)
As shown in Figure 1, the pivot-based approaches assume that the source-to-target parallel corpus Dx,z and the pivot-to-target parallel corpus Dz,y are available. As it is impractical to enumerate all possible pivot sentences, the two NMT models are trained separately in practice:
θ̂x→z = argmax θx→z { ∑ 〈x,z〉∈Dx,z logP (z|x;θx→z) }
θ̂z→y = argmax θz→y { ∑ 〈z,y〉∈Dz,y logP (y|z;θz→y) }
Due to the exponential search space of pivot sentences, the decoding process of translating an unseen source sentence x has to be divided into two steps:
ẑ = argmax z
{ P (z|x; θ̂x→z) } (3)
ŷ = argmax y
{ P (y|ẑ; θ̂z→y) } (4)
Despite its simplicity, the above two-step decoding process potentially suffers from the error propagation problem (Zhu et al., 2013): the translation errors made in the first step (i.e., source-topivot translation) will affect the second step (i.e., pivot-to-target translation).
Therefore, it is necessary to explore methods for direct modeling of source-to-target translation without parallel corpora available.
3 Approach
3.1 Assumptions
In this work, we propose to directly model the intended source-to-target neural translation based on a teacher-student framework. The basic idea is to use a pre-trained pivot-to-target model (“teacher”) to guide the learning process of a source-to-target model without training data available (“student”) on a source-pivot parallel corpus.
As shown in Figure 1(b), we still assume that a source-to-pivot parallel corpus Dx,z and a pivot-to-source parallel corpus Dz,y are available. Unlike pivot-based approaches, we first use the pivot-to-target parallel corpus Dz,y to obtain a teacher model P (y|z; θ̂z→y), where θ̂z→y is a set of learned model parameters. Then, the teacher model “teaches” the student model P (y|x;θx→y) based on the following assumption. Assumption 1 If a source sentence x is a translation of a pivot sentence z, then the probability of generating a target sentence y from x should be close to that from its counterpart z.
We can move a further step to introduce a wordlevel assumption:
Assumption 2 If a source word x is a translation of a pivot word z, then the probability of generating a target sentence y from x should be close to that from its counterpart z.
The two assumptions are empirically verified in our experiments (see Table 2). In the following subsections, we will introduce two approaches to zero-resource neural machine translation based on the two assumptions.
3.2 Sentence-Level Teaching
Given a source-pivot parallel corpus Dx,z , our training objective based on Assumption 1 is defined as follows:
JSENT(θx→y) = ∑ 〈x,z〉∈Dx,z KL ( P (y|z; θ̂z→y) ∣∣∣∣∣∣P (y|x;θx→y)) (5) where the KL divergence sums over all possible target sentences:
KL ( P (y|z; θ̂z→y) ∣∣∣∣∣∣P (y|x;θx→y)) =
∑ y∈Y P (y|z; θ̂z→y) log P (y|z; θ̂z→y) P (y|x;θx→y) (6)
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
As the teacher model parameters are fixed, the training objective can be equivalently written as
JSENT(θx→y) = − ∑ 〈x,z〉∈Dx,z Ey|z;θ̂z→y [ logP (y|x;θx→y) ] (7)
In training, our goal is to find a set of source-totarget model parameters that maximizes the training objective:
θ̂x→y = argmax θx→y
{ JSENT(θx→y) } (8)
However, a major difficulty is the intractability in calculating the gradients because of the exponential search space of target sentences. To address this problem, it is possible to construct a subspace by either sampling (Shen et al., 2016), generating a k-best list (Yong et al., 2016) or mode approximation (Kim and Rush, 2016). Then, standard stochastic gradient descent algorithms can be used to optimize model parameters.
3.3 Word-Level Teaching
Instead of minimizing the KL divergence between the teacher and student models at the sentence level, we further define a training objective based on Assumption 2 at the word level:
JWORD(θx→y) = ∑ 〈x,z〉∈Dx,z Ey|z;θ̂z→y [ J(x,y, z, θ̂z→y,θx→y) ] (9)
where
J(x,y, z, θ̂z→y,θx→y)
= |y|∑ j=1 KL ( P (y|z,y<j ; θ̂z→y) ∣∣∣∣∣∣ P (y|x,y<j ;θx→y) ) (10)
Equation (9) suggests that the teacher model P (y|z,y<j ; θ̂z→y) “teaches” the student model P (y|x,y<j ;θx→y) in a word-by-word way. Note that the KL-divergence between two models is defined at the word level:
KL ( P (y|z,y<j ; θ̂z→y) ∣∣∣∣∣∣P (y|x,y<j ;θx→y)) = ∑ y∈Vy P (y|z,y<j ; θ̂z→y) log P (y|z,y<j ; θ̂z→y) P (y|x,y<j ;θx→y)
Corpus Direction Train Dev. Test
Europarl Es→ En 850K 2,000 2,000 De→ En 840K 2,000 2,000 En→ Fr 900K 2,000 2,000
WMT Es→ En 6.78M 3,003 3,003 En→ Fr 9.29M 3,003 3,003
Table 1: Data statistics. For the Europarl corpus, we evaluate our approach on Spanish-French (EsFr) and Germany-French (De-Fr) translation tasks. For the WMT corpus, we our approach on the Spanish-French (Es-Fr) translation task. English is used a pivot language in all experiments.
Therefore, our goal is to find a set of source-totarget model parameters that maximizes the training objective:
θ̂x→y = argmax θx→y
{ JWORD(θx→y) } (11)
We use similar approach for approximating the full search space with sentence-level teaching.
4 Experiments
4.1 Setup
We evaluate our approach on the Europarl and WMT corpora. To compare with the pivot-based methods, we use the same dataset as in the paper (Cheng et al., 2016). The evaluation metric is case-insensitive BLEU (Papineni et al., 2002) as calculated by the multi-bleu.perl script. All the experiments treat English as the pivot language and French as the target language.
For the Europarl corpus (Koehn, 2005), we evaluate our proposed model on Spanish-French (Es-Fr) and Germany-French (De-Fr) translation tasks in a zero-resource scenario. To build nonoverlapping source-to-pivot and pivot-to-target datasets, we split pivot sentences shared by the original source-to-pivot and pivot-to-target corpora into two equal parts and merge them separately to the remaining source-to-pivot and pivotto-target corpora. Both the development and test datasets are from shared task 2006. All the sentences are tokenized by the tokenize.perl script. To deal with out-of-vocabulary words, we also adopt byte pair encoding (BPE) (Sennrich et al., 2016) to split words into sub-words. The size of sub-words is set to 30K for each language.
For the WMT corpus, we evaluate our approach on a Spanish-French (Es-Fr) translation task with
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Approx. P (y|x;θx→y) .
random init. trained
JSENT greedy 313.0 53.5 beam 323.5 51.8 JWORD greedy 274.0 37.1 beam 288.7 36.5 sampling 268.6 40.5
Table 2: Approximate KL divergence from source-to-target to pivot-to-target model. We use a random initialized source-to-target model for comparison.
a zero-resource setting. We combine the following corpora to form the Es-En and En-Fr parallel corpora: Common Crawl, News Commentary, Europarl v7 and UN. All the sentences are tokenized by the tokenize.perl script. Newstest2011 serves as the development set and newstest2012 and newstest2013 serve as test sets. We use case-sensitive BLEU to evaluate translation results. BPE is also used to reduce the vocabulary size. The size of sub-words is set to 43K, 33K, 43K for Spanish, English and French, respectively. See Table 1 for detailed statistics for both Europarl and WMT corpora.
We leverage a tutorial NMT open-source code implemented by Theano for all the experiments. 1 and compare our approach with state-of-the-art multilingual method (Firat et al., 2016b) and pivotbased method (Cheng et al., 2016) with the following variations:
1. Sentence-Level Teaching: for simplicity, we use mode as suggested in (Kim and Rush, 2016) to approximate the target sentence space, which is the result of running beam search on the pivot sentences and taking the highest-scoring target sentence with the teacher model. Beam size with K = 1 (greedy decoding) and K = 5 are investigated in our paper, denoted as sent-greedy and sent-beam, respectively.
2. Word-Level Teaching: we use the same mode approximation approach as in sentence-level teaching, denoted as word-greedy (beam search with K=1) and word-beam (beam search with K=5) respectively. Besides, Monte Carlo estimation by sampling from the
1dl4mt-tutorial: https://github.com/nyu-dl
teacher model is also investigated since it introduces more diverse data, denoted as wordsampling.
4.2 Assumptions Verification
To verify the assumptions in Section 3.1, we train a source-to-target translation model P (y|x;θx→y) and a pivot-to-target translation model P (y|z;θz→y) using trilingual corpus. Then we use these two models to calculate JSENT and JWORD, which reveals how close the sentencelevel and word-level distributions are, respectively. Since it is difficult to measure JSENT and JWORD directly , we instead calculate their approximations as illustrated in Section 4.1. We report average KL divergence with different approximations evaluated on the aligned trilingual development set Exzy with 2000 sentences from shared task 2006 as shown in Table 1. We also report KL Divergence from a randomly initialized NMT model Prandom to pivot-to-target P (y|z,θz→y) for comparison.
The results agree with our assumptions since significantly smaller KL divergence is observed for KL ( P (y|z;θz→y)||P (y|x;θx→y) ) . For sentence-level distributions, smaller KL divergence for beam search approximation is observed compared with greedy approximation, indicating that mode produced by the beam search results of P (y|z;θz→y) have higher probabilities in P (y|x;θx→y) compared with the greedy results. Thus, we suspect that the sent-beam method will surpass the sent-greedy method. For word-level results, KL divergence of sampling approximation is the highest, indicating poorer performance than the other two methods. However, sampling methods introduce more data diversity at the target side. Thus it is harder to decide which factor dominates the training process. We leave further discussion to Section 4.3.
4.3 Results on the Europarl Corpus
4.3.1 Comparison with pivot-based methods
Table 3 gives BLEU scores on the Europarl corpus of our best performing sentence-level method (sent-beam) and word-level method (wordsampling) compared with pivot-based methods (Cheng et al., 2016). The same data preprocessing are applied in (Cheng et al., 2016) as in our experiments. We find that our sentence-level method and word-level method outperform all of their zero-resource approaches across language
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Method BLEU
Es→ Fr De→ Fr
Cheng.# pivot 29.79 23.70 hard 29.93 23.88 soft 30.57 23.79 likelihood† 32.59 25.93 Ours sent-beam 31.64 24.39 word-sampling 33.86 27.03
Table 3: Comparison with previous work on Spanish-French and German-French translation tasks from Europarl corpus. English is treated as the pivot language. † denotes using parallel source-to-target corpus. # denotes methods in Cheng et al. (2016).
pairs. Our word-sampling method improves their best zero-resource results on Spanish-French translation by 3.29 BLEU and Germany-French translation by 3.15 BLEU. Besides that, our word-sampling method surprisingly obtains improvement over the likelihood method, which leverages source-to-target parallel corpus and also takes much longer to train. The significant improvements can be explained by the error propagation problem of pivot-based methods that translation error of the source-pivot translation system will be transferred to the pivot-target translation.
4.3.2 Comparison of Sentence-Level and Word-Level Methods
Table 4 shows BLEU scores on the Europarl corpus of our proposed five methods.
For sentence-level approaches, sent-beam outperforms sent-greedy by 0.59 BLEU over es→ fr translation and 2.51 BLEU over de→ fr translation. The results are in line with our observations in Section 4.2 that sentence-level KLD by beam approximation is smaller than that by greedy approximation. However, as the time complexity grows linearly in the number of beams k, the better performance is achieved at the expense of beam search time.
For word-level experiments, we observe that word-sampling performs much better than the other two methods: 1.94 BLEU on es→ fr translation and 2.65 BLEU on de→ fr translation; 1.88 BLEU on es→ fr translation and 2.84 BLEU on de→ fr translation. Althrough word-level KLD calculated by Monte Carlo estimation is larger
Methods #sents
BLEU de-en en-fr de-fr
MLE − − 840K 26.06 - - 1.5M 29.02 Ours 840K 900K - 27.03
Table 5: Comparison on German-French translation tasks from Europarl corpus with stardard NMT.
than that by mode approximation, word-sampling introduces more data diversity for training, which dominates the effect of KLD difference.
4.3.3 BLEU Score over Iterations
Figure 3 shows valid loss 2 and BLEU score over iterations. We observe that word-level models tend to have lower valid loss compared with sentencelevel methods. Generally, models with lower valid loss are inclined to have higher BLEU. Our results indicate that this is not necessarily the case: sentbeam converges to +0.82 the BLEU on validation set with -13 valid loss compared with the wordbeam method. Kim and Rush (2016) claims a similar observation in data distillation for NMT and provides an explanation that student distributions are more peaked for sentence-level models. This is indeed the case in our result: on Germany-French translation task the argmax for the sent-beam student model (on average) accounts approximately for 3.49% of the total probability mass, while the corresponding number is 1.25% for the wordbeam student model and 2.60% for the teacher model. However, argmax for the sentence-level model in (Kim and Rush, 2016) comprises around 15% output probability, while ours only accounts for 3.49%. It possibly reveals that it is more diffi-
2Valid loss: the average NLL of sentence pairs on the validation set.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Method # Training Sents
# Para. BLEU
Es→ En En→ Fr Newstest2012 Newstest2013 Existing zero-resource NMT systems
Cheng et al. (2016)† pivot 6.78M 9.29M - 19.81 - Cheng et al. (2016)† likelihood 6.78M 9.29M 100K 21.48 - Firat et al. (2016b) one-to-one 34.71M 65.77M - 17.59 17.61 Firat et al. (2016b)† many-to-one 34.71M 65.77M - 21.33 21.19
Our zero-resource NMT systems word-sampling 6.78M 9.29M - 25.26 25.06
Table 6: Comparison with previous work on Spanish-French translation in zero resource scenario over WMT corpus. The BLEU scores are case sensitive. †: the pivot path involved in the translation path during testing.
cult to model around the teacher’s mode for crosstask teaching than in data distillation.
4.3.4 Comparison with Standard NMT
Table 5 gives the comparison on German-French zero-resource translation between word-sampling approach and standard NMT model trained by MLE. Our proposed method even slightly outperforms standard NMT system trained with parallel source-to-target corpus when this corpus contains similar number of sentences as source-topivot corpus, demonstrating that the effectiveness of our methods to learn from teacher model. This phenomenon can be explained by that the total size of corpus involved in our method is twice that of standard MLE. The translation knowledge contained in other language pairs benefit source-totarget translation, as observed in multilingual scenario (Johnson et al., 2016; Ha et al., 2016).
4.3.5 BLEU Score with Low-Resource Source-to-Pivot Corpus
Our proposed method can also be applied to zeroresource NMT with low source-to-pivot corpus. Specifically, the size of source-pivot corpus is magnitude smaller than that of pivot-target corpus. This setting makes sense in applications. For example, there are significantly fewer Urdu-English corpus available than English-French corpus.
To fulfill this task, we combine our best performing word-sampling method with the initialization and parameter freezing strategy proposed in (Zoph et al., 2016). We set the size of DeEn corpus to be 100K and use the same teacher model trained with 900K corpus. Table 7 gives the BLEU score of our method on De-Fr translation compared with three other methods. Noting that our task is much harder than (Zoph et al.,
2016) (transfer) since they are given parallel DeFr corpus directly. Surprisingly, our method outperforms all other methods. We significantly improve the baseline pivot method by 5.63 BLEU and the state-of-the-art low resource method transfer by 0.56 BLEU. We leave further evaluation of this method for future work. 3
4.4 Results on the WMT Corpus
Word-sampling method obtains the best performance in our five proposed approaches according to experiments on the Europarl corpus. To further investigate this approach, we conduct experiment on large scale WMT corpus for es→ fr translation. Table 6 shows the results of our word-sampling method in comparison with other state-of-the-art baselines. Cheng et al.(2016) use the same training&develpment sets and the same
3We test initialization trick in our original settings and observe that with initialization, the BLEU starts to increase very fast (21.25 Bleu with 10000 iterations while without initialization, the number is 2.5). However the model converges to similar point as without initialization.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
0 3 6 9 1 2 1 53 0
6 0
9 0
1 2 0
1 5 0
1 8 0
2 1 0
0 3 6 9 1 2 1 50
5
1 0
1 5
2 0
2 5 3 0 V a lid L o ss
I t e r a t i o n s
s e n t  g r e e d y s e n t  b e a m w o r d  g r e e d y w o r d  b e a m w o r d  s a m p l i n g
×1 04 ×1 04
B L
E U
I t e r a t i o n s
s e n t  g r e e d y s e n t  b e a m w o r d  g r e e d y w o r d  b e a m w o r d  s a m p l i n g
Figure 2: Validation loss and BLEU across iterations of our proposed methods.
preprocessing as ours. Firat et al.(2016b) uses a much larger training sets. Our method obtains significant improvement over the pivot baseline by 5.45 BLEU on newstest2012 and over many-toone 3.87 BLEU on newstest2013, not to mention that both method are depending on source-pivottarget decoding path.
5 Related Work
Training NMT system in zero-resource scenario by leveraging some other languages has attracted intensive attention in last year. Firat et al. (2016b) propose an approach which genuinely delivers the multi-way, multilingual NMT model proposed by (Firat et al., 2016a) to zero-resource translation. They use the multi-way NMT model trained by other language pairs to generate pseudo parallel corpus and fine-tune the attention mechanism of the multi-way NMT model to enable zero-resource translation. Several authors propose a universal encoder-decoder network in multilingual scenarios to perform zero-shot learning. This universal model extracts translation knowledge from multiple different languages, making zero-resource translation feasible without directly training (Johnson et al., 2016; Ha et al., 2016).
Besides multilingual NMT, another import line of research is pivot-based methods. Inspired by the widely used pivot idea in SMT (De Gispert and Marino, 2006; Cohn and Lapata, 2007; Utiyama and Isahara, 2007; Wu and Wang, 2007; Bertoldi et al., 2008; Zahabi et al., 2013; Kholy et al., 2013), Cheng et al.(2016) propose pivotbased NMT by simultaneously improving source
to pivot and pivot to target translation quality in order to improve source to target translation quality. Nakayama and Nishida (2016a) achieve zeroresource machine translation by utilizing image as pivot.
In (Kim and Rush, 2016), the author first introduce knowledge distilling in neural machine translation. They suggest to generate pseudo corpus to train the student network. Our work is highly related to theirs. However, we focus on zeroresource learning instead of model compression.
6 Conclusion
In this paper, we propose a novel framework to transfer the knowledge of the teacher model trained with rich-resource language pair into the student model with zero-resource with the help of source-to-pivot corpus. We introduce sentencelevel and word-level teaching to guide the learning process of the student model. Our experiments on Europarl Corpus and WMT corpus across languages show that our proposed word-level sampling method can significantly outperforms the state-of-the-art pivot-based methods and multilingual methods in terms of both translation quality and decoding efficiency.
We also analyze zero-resource translation with low source-to-pivot corpus, and combine our word-level sampling method with initialization and parameter freezing suggested by (Zoph et al., 2016). The experiments on Europarl Corpus show that our approach obtains an significant improvement over pivot method significantly.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This paper proposes a novel strategy for zero-resource translation where (source, pivot) and (pivot, target) parallel corpora are available. A teacher model for p(target|pivot) is first trained on the (pivot, target) corpus, then a student model for p(target|source) is trained to minimize relative entropy with respect to the teacher on the (source, pivot) corpus. When using word-level relative entropy over samples from the teacher, this approach is shown to outperform previous variants on standard pivoting, as well as other zero-resource strategies.  This is a good contribution: a novel idea, clearly explained, and with convincing empirical support. Unlike some previous work, it makes fairly minimal assumptions about the nature of the NMT systems involved, and hence should be widely applicable.  I have only a few suggestions for further experiments. First, it would be interesting to see how robust this approach is to more dissimilar source and pivot languages, where intuitively the true p(target|source) and p(target|pivot) will be further apart. Second, given the success of introducing word-based diversity, it was surprising not to see a sentence n-best or sentence-sampling experiment. This would be more costly, but not much more so since you re already doing beam search with the teacher. Finally, related to the previous, it might be interesting to explore transition from word-based diversity to sentence-based as the student converges and no longer needs the signal from low-probability words.  Some further comments:  line 241: Despite its simplicity to Due to its simplicity  277: target sentence y to target word y  442: I assume that K=1 and K=5 mean that you compare probabilities of the most probable and 5 most probable words in the current context. If so, how is the current context determined - greedily or with a beam?  Section 4.2. The comparison with an essentially uniform distribution doesn t seem very informative here: it would be extremely surprising if p(y|z) were not significantly closer to p(y|x) than to uniform. It would be more interesting to know to what extent p(y|z) still provides a useful signal as p(y|x) gets better. This would be easy to measure by comparing p(y|z) to models for p(y|x) trained on different amounts of data or for different numbers of iterations. Another useful thing to explore in this section would be the effect of the mode approximation compared to n-best for sentence-level scores.  555: It  odd that word beam does worse than word greedy, since word beam should be closer to word sampling. Do you have an explanation for this?  582: The claimed advantage of sent-beam here looks like it may just be noise, given the high variance of these curves.


End-to-end automatic speech recognition (ASR) has become a popular alternative to conventional DNN/HMM hybrid systems because it avoids the need for linguistic resources such as pronunciation dictionary, tokenization, and contextdependency trees, leading to a greatly simplified model-building process. There are two major types of end-to-end architectures for ASR: attention-based methods use an attention mechanism to perform alignment between acoustic frames and recognized symbols, and connectionist temporal classification (CTC), uses Markov assumptions to efficiently solve sequential problems by dynamic programming. This paper proposes joint CTCattention end-to-end ASR, which effectively utilizes both advantages in training and decoding. We have applied the proposed method to two ASR benchmarks (spontaneous Japanese and Mandarin Chinese), and showing the comparable performance to conventional hybrid ASR systems without linguistic resources.
1 Introduction
Automatic speech recognition (ASR) is currently a mature set of technologies that have been widely deployed, resulting in great success in interface applications such as voice search. A typical ASR system is factorized into several modules including acoustic, lexicon, and language models based on a probabilistic noisy channel model (Jelinek, 1976). Over the last decade, dramatic improvements in acoustic and language models have been driven by machine learning techniques known as deep learning (Hinton et al., 2012). However,
current systems lean heavily on the scaffolding of complicated legacy architectures that grew up around traditional techniques. These present the following problems that we may seek to eliminate.
1. Flat start: many module-specific processes are required to build an accurate module: for example, when we build an acoustic model from scratch, we have to first build hidden Markov model (HMM) and Gaussian mixture model (GMM) followed by deep neural networks (DNN). 2. Linguistic knowledge: to well factorize acoustic and language models, we need to have a lexicon model, which is usually based on a hand-crafted pronunciation dictionary to map word to phoneme sequence. Also, some languages do not explicitly have a word boundary and need tokenization modules (Kudo et al., 2004; Bird, 2006). 3. Conditional independence assumptions: the current ASR systems often use conditional independence assumptions (especially Markov assumptions) during the above factorization and to make use of GMM, DNN, and n-gram models. The data do not necessarily follow such assumptions leading to model mis-specification. 4. Complex decoding: inference/decoding has to be performed by integrating all modules. Although this integration is often efficiently handled by finite state transducers, the construction and implementation of welloptimized transducers is very complicated. 5. Local optimum: the above modules are optimized separately, which may result in local optima, where each module is trained to match the other modules1.
1 Sequence discriminative training (Veselỳ et al., 2013) can solve the issue to some extent, but requires additional
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Consequently, it is quite difficult for non-experts to use/develop ASR systems for new applications, especially for new languages.
End-to-end ASR has the goal of simplifying the above module-based architecture into a singlenetwork architecture within a deep learning framework, in order to address the above issues. There are two major types of end-to-end architectures for ASR: attention-based methods use an attention mechanism to perform alignment between acoustic frames and recognized symbols, and connectionist temporal classification (CTC), uses Markov assumptions to efficiently solve sequential problems by dynamic programming (Chorowski et al., 2014; Graves and Jaitly, 2014).
All ASR models aim to elucidate the posterior distribution p(W |X) of word sequence W given speech feature sequence X . The end-to-end methods do this directly whereas conventional models factorize p(W |X) into modules such as the language model p(W ), which can be trained on pure language data, and an acoustic model likelihood p(X|W ) which is trained on acoustic data with the corresponding language labels. End-to-end ASR methods typically rely only on paired acoustic and language data. Without the additional language data, they can suffer from data sparseness or outof-vocabulary issues. To improve generalization, and handle out-of-vocabulary problems, it is typical to use the letter representation rather than the word representation for the language output sequence, which we adopt in the descriptions below.
The attention-based end-to-end method solves the ASR problem as a sequence mapping from speech feature sequences to text by using encoderdecoder architecture. The decoder network uses an attention mechanism to find an alignment between each element of the output sequence and the hidden states generated by the acoustic encoder network for each frame of acoustic input (Chorowski et al., 2014, 2015; Chan et al., 2015; Lu et al., 2016). At each output position, the decoder network computes a matching score between its hidden state and the states of the encoder network at each input time, to form a temporal alignment distribution, which is then used to extract an average of the corresponding encoder states.
This basic temporal attention mechanism is too flexible in the sense that it allows extremely non-
process in the above modules including lattice generations.
sequential alignments. This may be fine for applications such as machine translation where input and output word order are different (Bahdanau et al., 2014; Wu et al., 2016). However in speech recognition, the feature inputs and corresponding letter outputs generally proceed in the same order, with only small within-word deviations (e.g. the word ”iron” which transposes the sounds for ”r” and ”o”). Another problem is that the input and output sequences in ASR can have very different lengths, and these vary greatly from case to case, depending on the speaking rate and writing system, making it more difficult to track the alignment.
However, an advantage is that the attention mechanism does not require any conditional independence assumptions, and could address all the problems cited above. Although the alignment problems of attention-based mechanisms have been partially addressed in (Chorowski et al., 2014; Chorowski and Jaitly, 2016) using various mechanisms, here we propose more rigorous constraints by using CTC-based alignment to guide the training.
CTC permits an efficient computation of a strictly monotonic alignment using dynamic programming (Graves et al., 2006; Graves and Jaitly, 2014) although it requires language models and graph-based decoding (Miao et al., 2015) except in the case of huge training data (Amodei et al., 2015; Soltau et al., 2016). We propose to take advantage of the constrained CTC alignment in a hybrid CTC-attention based system. During training, we attach a CTC objective to an attention-based encoder network as a regularization, as proposed by (Kim et al., 2016). This greatly reduces irregular alignments without any heuristic search techniques. During decoding, we propose to use a rescoring technique, where hypotheses of attentionbased ASR are refined by scores obtained by using encoder outputs.
The proposed method is applied to Japanese and Mandarin ASR tasks, which require extra linguistic resources including morphological analyzer (Kudo et al., 2004) or word segmentation (Xue et al., 2003) in addition to pronunciation dictionary to provide accurate lexicon and language models in conventional hybrid ASR. Surprisingly, the method achieved performance comparable to, and in some cases superior to, several state-of-theart hybrid ASR systems, without using the above
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
linguistic resources.
2 From hybrid to end-to-end ASR
This section briefly provides a formulation of conventional HMM/DNN hybrid ASR and CTC or attention based end-to-end ASR. The formulation is intended to clarify the conditional independence assumption (Markov assumption), which is an important property to characterize these three methods.
2.1 Hybrid HMM/DNN
ASR deals with a sequence mapping from T - length speech feature sequence X = {xt ∈ RD|t = 1, · · · , T} to N -length word sequence W = {wn ∈ V|n = 1, · · · , N}. xt is a D dimensional speech feature vector (e.g., log Mel filterbanks) at frame t and wn is a word at position n in vocabulary V .
ASR is mathematically formulated with the Bayes decision theory, where the most probable word sequence Ŵ is estimated among all possible word sequences V∗ as follows:
Ŵ = arg max W∈V∗
p(W |X). (1)
Therefore, the main problem of ASR is how to obtain the posterior distribution p(W |X).
In the current main stream of ASR is called hybrid HMM/DNN (Bourlard and Morgan, 1994), which uses the Bayes theorem and introduces HMM state sequence S = {st ∈ {1, · · · , J}|t = 1, · · · , T} to factorize p(W |X) into the following three distributions:
arg max W
p(W |X)
≈ arg max W ∑ S p(X|S)p(S|W )p(W ). (2)
The three factors, p(X|S), p(S|W ), and p(W ), are acoustic, lexicon, and language models, respectively. Eq. (2) is obtained by a conditional independence assumption (i.e., p(X|S,W ) ≈ p(X|S)), which is a reasonable assumption to simplify the dependency of the acoustic model.
Acoustic model p(X|S) p(X|S) is further factorized by using a probabilistic chain rule and conditional independence assumption as follows:
p(X|S) ≈ ∏ t p(xt|st) ∝ ∏ t p(st|xt) p(st) , (3)
where the framewise likelihood function p(xt|st) is replaced with the framewise posterior distribution p(st|xt)/p(st) computed by powerful DNN classifiers by using so-called pseudo likelihood trick (Bourlard and Morgan, 1994). The conditional independence assumption in Eq. (3) is often regarded as too strong assumption, since it does not consider any input and hidden state contexts. Therefore DNNs with long context features or recurrent neural networks are often used to mitigate the issue. To train the framewise posterior, we also require to provide a framewise state alignment st as a target, which is often provided by a HMM/GMM system.
Lexicon model p(S|W ) p(S|W ) is also factorized by using a probabilistic chain rule and conditional independence assumption (1st-order Markov assumption) as follows:
p(S|W ) ≈ ∏ t p(st|st−1,W ) (4)
This probability is represented by an HMM state transition given W . The conversion from W to HMM states is deterministically performed by using a pronunciation dictionary through a phoneme representation.
Language model p(W ) Similarly, p(W ) is factorized by using a probabilistic chain rule and conditional independence assumption (m − 1th-order Markov assumption) as a m-gram model, i.e., p(W ) ≈∏
n p(wn|wn−1, . . . , wn−m−1). Although recurrent neural network language models (RNNLMs) can avoid this conditional independence assumption issue (Mikolov et al., 2010), it makes the decoding complex, and RNNLMs are often combined with m-gram language models based on as a re-scoring technique.
Thus, conventional hybrid HMM/DNN systems make the ASR problem formulated with Eq. (1) feasible by using factorization and conditional independence assumptions, at the cost of the five problems discussed in Section 1.
2.2 Connectionist Temporal Classification (CTC)
The CTC formulation also follows from Bayes decision theory (Eq. (1)). Note that the CTC formulation uses L-length letter sequence C = {cl ∈ U|l = 1, · · · , L} with a set of distinct letters U .
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Similarly to Section 2.1, by introducing framewise letter sequence with an additional ”blank” symbol Z = {zt ∈ U ∪ blank|t = 1, · · · , T}, and by using the probabilistic chain rule and conditional independence assumption, the posterior distribution p(C|X) is factorized as follows:
p(C|X) ≈ ∑ Z p(C|Z)p(Z|X) ≈ ∑ Z p(C|Z) ∏ t p(zt|X)
≈ ∑ Z ∏ t
p(zt|zt−1, C)p(zt|X)︸ ︷︷ ︸ ,pctc(C|X) p(C) (5)
CTC is similar to the hybrid approach, except that it applies Bayes theorem to p(C|Z) instead of to p(W |X). As a result, CTC has three distribution components similar to the hybrid case, i.e., framewise posterior distribution p(zt|X), transition probability p(zt|zt−1, C), and letter-based language model p(C). We also define the CTC objective function pctc(C|X) used in the later formulation.
The framewise posterior distribution p(zt|X) is conditioned on all inputs X , and it is quite natural to be modeled by using bidirectional long shortterm memory (BLSTM):
p(zt|X) = Softmax(Lin(ht)) (6) ht = BLSTM(X). (7)
Softmax(·) is a sofmax activation function, and Lin(·) is a linear layer to convert hidden vector ht to a (|U| + 1) dimensional vector (+1 means a blank symbol introduced in CTC).
Although Eq. (5) has to deal with a summation over all possible Z, it is efficiently computed by using dynamic programming (Viterbi/forwardbackward algorithm) thanks to the Markov property. In summary, although CTC and hybrid systems are similar to each other due to conditional independence assumptions, CTC does not require pronunciation dictionaries and omits an HMM/GMM construction step.
2.3 Attention mechanism
Compared with hybrid and CTC approaches, the attention-based approach does not make any conditional independence assumptions, and directly estimates the posterior p(C|X) based on a prob-
abilistic chain rule, as follows: p(C|X) = ∏ l
p(cl|c1, · · · , cl−1, X)︸ ︷︷ ︸ ,patt(C|X) , (8)
where patt(C|X) is an attention-based objective function. p(cl|c1, · · · , cl−1, X) is obtained by
p(cl|c1, · · · , cl−1, X) = Decoder(rl,ql−1, cl−1) ht = Encoder(X) (9)
alt = Attention({al−1}t,ql−1,ht) (10) rl = ∑ t altht. (11)
Eq. (9) converts input feature vectors X into a framewise hidden vector ht in an encoder network based on BLSTM, i.e., Encoder(X) , BLSTM(X). Attention(·) in Eq. (10) is based on a content-based attention mechanism with convolutional features, as described in (Chorowski et al., 2015) (see Appendix B). alt is an attention weight, and represents a soft alignment of hidden vector ht for each output cl based on the weighted summation of hidden vectors to form letter-wise hidden vector rl in Eq. (11). A decoder network is another recurrent network conditioned on previous output cl−1 and hidden vector ql−1, similar to RNNLM, in addition to letter-wise hidden vector rl. We use Decoder(·) , Softmax(Lin(LSTM(·))).
Attention-based ASR does not explicitly separate each module, and potentially handles the all five issues pointed out in Section 1. It implicitly combines acoustic models, lexicon, and language models as encoder, attention, and decoder networks, which can be jointly trained as a single deep neural network.
Compared with hybrid and CTC, which are based on a transition form from t − 1 to t due to the Markov assumption, the attention mechanism does not maintain this constraint, and often provides irregular alignments. A major focus of this paper is to address this problem by proposing joint CTC-attention models.
3 Joint CTC-attention
This section explains a joint CTC-attention network, which utilizes both benefits of CTC and attention during training and decoding steps in ASR.
3.1 Multi-task learning
Kim et al. (2016) uses a CTC objective function as an auxiliary task to train the attention model en-
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
ㅡ ㅡㅡz2 …
sos eosc1
q0
r0 r1 rL
H
h2 h4
q1 qL
hT
x1 x2 x3 x4 x5 x6 xT
Shared Encoder
CTC
Attention Decoder
x7 x8
z4
h6 h8
…
c2
r2
q2
…c1 c2 …
c1 c2 …
Figure 1: Joint CTC-attention based end-to-end framework: the shared encoder is trained by both CTC and attention model objectives simultaneously. The shared encoder transforms our input sequence {xt · · ·xT } into high level featuresH = {ht · · ·hT }, and the attention decoder generates the letter sequence {c1 · · · cL}.
coder within the multi-task learning (MTL) framework. Figure 1 illustrates the overall architecture of the framework, where the same BLSTM is shared with CTC and attention encoder networks (that is Eqs. (7) and (9), respectively). Unlike the sole attention model, the forward-backward algorithm of CTC can enforce monotonic alignment between speech and label sequences during training. That is, rather than solely depending on data-driven attention methods to estimate the desired alignments in long sequences, the forwardbackward algorithm in CTC helps to speed up the process of estimating the desired alignment. The objective to be maximized is a logarithmic linear combination of the CTC and attention objectives, i.e., pctc(C|X) in Eq. (5) and patt(C|X) in Eq. (8):
LMTL = λ log pctc(C|X) + (1− λ) log patt(C|X), (12)
with a tunable parameter λ : 0 ≤ λ ≤ 1.
3.2 Joint CTC-attention decoding
The inference step of our joint CTC-attention based end-to-end speech recognition is performed by output-label synchronous decoding with a beam search like attention-based ASR. But, we take the CTC probabilities into account to find a better aligned hypothesis to the input speech, as shown in Figure 1
End-to-end speech recognition inference is generally defined as a problem to find the most probable letter sequence Ĉ given speech input X , i.e.
Ĉ = arg max C∈U∗
log p(C|X). (13)
In attention-based ASR, p(C|X) is computed by Eq. (8), and Ĉ is found by the beam search.
Let Ωl be a set of partial hypotheses of length l. At the beginning of the beam search, Ω0 contains only one hypothesis with starting symbol <sos>. For l = 1 to Lmax, each partial hypothesis in Ωl−1 is expanded by appending possible single letters, and the new hypotheses are stored in Ωl, where Lmax is the maximum length of hypotheses to be searched. The score of each new hypothesis is computed in log domain as
α(gl−1 · c) = α(gl−1) + log p(c|gl−1, X), (14)
where gl−1 is a partial hypothesis in Ωl−1 and c is a letter appended to gl−1. The new partial hypothesis gl, i.e. gl−1 · c, is added to Ωl. If c is a special symbol that represents the end of sequence, <eos>, gl is added to Ω̂ instead of Ωl, where Ω̂ denotes a set of complete hypotheses. Finally, Ĉ is obtained as
Ĉ = arg max g∈Ω̂ α(g). (15)
In the beam search, Ωl is allowed to hold only a limited number of hypotheses with higher scores to improve the search efficiency.
Attention-based ASR, however, may be prone to deletion and insertion errors (see Appendix A) because of its flexible alignment property, which can attend any portion of the encoder state sequence to predict the next label, as discussed in Section 2.3. Since the attention is generated by the decoder network it may prematurely predict the end-of-sequence label, even when it has not attended all the encoder frames, making the hypothesis too short. On the other hand, it may predict the next label with a high probability by attending the same portions as those attended before. In this case, the hypothesis becomes very long and includes repetitions of the same label sequence.
To alleviate this problem, a length penalty term is commonly used to control the hypothesis length to be selected (Chorowski et al., 2015; Bahdanau et al., 2016). With the length penalty, the decoding objective is changed to
Ĉ = arg max C∈U∗
{log p(C|X) + γ|C|} , (16)
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
where |C| is the length of sequence C and γ is a tunable parameter. However, it is actually difficult to completely exclude too short and too long hypotheses even if γ is carefully tuned. It is also effective to control the hypothesis length by minimum and maximum lengths to some extent, where the minimum and the maximum are selected as fixed rations to the length of input speech. But, since there are exceptionally long or short transcripts compared to the input speech, it is difficult to balance saving such exceptional transcripts and preventing hypotheses with irrelevant lengths.
Another approach is a coverage term recently proposed in (Chorowski and Jaitly, 2016), which is incorporated in the decoding objective as
Ĉ = arg max C∈U∗
{log p(C|X) + γ|C|
+η · coverage(C|X)} , (17)
and computed by
coverage(C|X) = T∑ t=1 [ L∑ l=1 alt > τ ] , (18)
where η and τ are tunable parameters. The coverage term represents the number of frames that have received a cumulative attention greater than τ . Accordingly, it increases when paying close attention to some frames for the first time, but does not increase when paying attention again to the same frames. This property is effective to avoid looping the same label sequence within a hypothesis. However, it is still difficult to obtain a common parameter setting for γ, η, τ and optional min/max lengths so that they are appropriate for any speech data from different tasks.
Our joint CTC-attention approach combines CTC and attention-based sequence probabilities in the inference step, as well as the training step. Suppose pctc(C|X) in Eq. (5) and patt(C|X) in Eq. (8) are the sequence probabilities given by CTC and the attention model, respectively. The decoding objective is defined similarly to Eq. (12) as
Ĉ = arg max C∈U∗
{λ log pctc(C|X)
+(1− λ) log patt(C|X)} . (19)
The CTC probability enforces a monotonic alignment that does not allow big jumps or looping the same frames. Accordingly, it is possible to
choose a hypothesis with a better alignment and exclude irrelevant hypotheses without relying on any of the coverage term, the length penalty, and the min/max lengths.
We implement the joint CTC-attention inference method as a two-pass strategy. The first pass obtains a set of complete hypotheses using the beam search, where only attention-based sequence probabilities are considered. The second pass rescores the complete hypotheses using the CTC-attention probabilities, where the CTC probabilities are obtained by the forward algorithm for CTC (Graves et al., 2006). The rescoring pass obtains the final result according to
Ĉ = arg max g∈Ω̂ {(1− λ)α(g) + λξ(g)} , (20)
where ξ(g) = log pctc(g|X).
4 Experiments
We used Japanese and Mandarin Chinese ASR benchmarks to show the effectiveness of the proposed joint CTC-attention approach. The main reason for choosing these two languages is that those ideogram languages have relatively shorter lengths for letter sequences than those in alphabet languages, which reduces computational complexities greatly, and makes it easy to handle context information in a decoder network. Our preliminary investigation shows that Japanese and Mandarin Chinese end-to-end ASR can be easily scaled up, and shows state-of-the-art performance without using various tricks developed in English tasks.
4.1 Corpus of Spontaneous Japanese (CSJ)
We demonstrated ASR experiments by using the Corpus of Spontaneous Japanese (CSJ) (Maekawa et al., 2000). CSJ is a standard Japanese ASR task based on a collection of monologue speech data including academic lectures and simulated presentations. It has a total of 581 hours of training data and three types of evaluation data, where each evaluation task consists of 10 lectures (totally 5 hours). As input features, we used 40 melscale filterbank coefficients, with their first and second order temporal derivatives to obtain a total of 120-dimensional feature vector per frame. The encoder was a 4-layer BLSTM with 320 cells in each layer and direction, and linear projection layer is followed by each BLSTM layer. The
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Table 1: Character error rate (CER) for conventional attention and proposed joint CTC-attention end-toend ASR. Corpus of Spontaneous Japanese speech recognition (CSJ) task.
Model Hour Task1 Task2 Task3 Attention 581 11.4 7.9 9.0 MTL 581 10.5 7.6 8.3 MTL + joint decoding 581 10.1 7.1 7.8 MTL-large + joint decoding 581 9.4 7.0 7.5 GMM-discr. (Moriya et al., 2015) 236 for AM, 581 for LM 11.2 9.2 12.1 DNN-hybrid (Moriya et al., 2015) 236 for AM, 581 for LM 9.0 7.2 9.6 CTC-syllable (Kanda et al., 2016) 581 9.4 7.3 7.5
2nd and 3rd bottom layers of the encoder read every second hidden state in the network below, reducing the utterance length by the factor of 4. We used the content-based attention mechanism (Chorowski et al., 2015), where the 10 centered convolution filters of width 100 were used to extract the convolutional features. The decoder network was a 1-layer LSTM with 320 cells. The AdaDelta algorithm (Zeiler, 2012) with gradient clipping (Pascanu et al., 2012) was used for the optimization. The joint CTC-attention ASR was implemented by using the Chainer deep learning toolkit (Tokui et al., 2015).
Table 1 first compares the character error rate (CER) for conventional attention and MTL based end-to-end ASR without the joint decoding. λ in Eq. (12) was set to 0.1. When decoding, we manually set the minimum and maximum lengths of output sequences by 0.1 and 0.5 times input sequence lengths, respectively. The length penalty γ in Eq. (16) was set to 0.1. MTL significantly outperformed attention-based ASR in the all evaluation tasks, which confirms the effectiveness of joint CTC-attention model. Table 1 also shows that the joint decoding, described in Section 3.2, further improved the performance without setting any search parameters (maximum and minimum lengths, length penalty), but only setting a weight parameter λ = 0.1 in Eq. (20) similar to the MTL case. Figure 2 also compares the dependency of λ on the CER for the CSJ evaluation tasks, and showing that λ was not so sensitive to the performance if we set λ around the value we used at MTL (i.e., 0.1).
We also compare the performance of the proposed MTL-large, which has a larger network (5-layer encoder network), with the conventional state-of-the-art techniques obtained by using linguistic resources. The state-of-the-art CERs of GMM discriminative training and DNN-sMBR
6.0
7.0
8.0
9.0
10.0
11.0
12.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
CE R  (% )
Re‐scoring weight
Task1 Task2 Task3
Figure 2: The effect of re-scoring weight λ in Eq. (20) on the CSJ evaluation tasks.
hybrid systems are obtained from the Kaldi recipe (Moriya et al., 2015) and a system based on syllable-based CTC with MAP decoding (Kanda et al., 2016). The Kaldi recipe systems use academic lectures (236h) for AM training and all training-data transcriptions for LM training. Unlike the proposed method, these methods use linguistic resources including a morphological analyzer, pronunciation dictionary, and language model. Note that since the amount of training data and experimental configurations of the proposed and reference methods are different, it is difficult to compare the performance listed in the table directly. However, since the CERs of the proposed method are comparable to those of the best reference results, we can state that the proposed method reaches the state-of-the-art performance.
4.2 Mandarin telephone speech
We demonstrated ASR experiments on HKUST Mandarin Chinese conversational telephone speech recognition (MTS) (Liu et al., 2006). It has 5 hours recording for evaluation, and we extracted 5 hours from training data as a development set, and used the rest (167 hours) as a training set. All experimental conditions were same as those in Section 4.1 except that we used the λ = 0.5 in training and decoding instead of
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Table 2: Character error rate (CER) for conventional attention and proposed joint CTC-attention end-toend ASR. HKUST Mandarin Chinese conversational telephone speech recognition (MTS) task.
Model dev eval Attention 40.3 37.8 MTL 38.7 36.6 Attention + coverage 39.4 37.6 MTL + coverage 36.9 35.3 MTL + joint decoding 35.9 34.2 MTL + joint decoding (speed perturb.) 32.1 31.4 DNN-hybrid – 35.9 LSTM-hybrid (speed perturb.) – 33.5 CTC with language model (Miao et al., 2016) – 34.8 TDNN-hybrid, lattice-free MMI (speed perturb.) (Povey et al., 2016) – 28.2
0.1 based on our preliminary investigation and 80 mel-scale filterbank coefficients with pitch feature as an input by referring (Miao et al., 2016). In decoding, we also added a result of the coverage-term based decoding (Chorowski and Jaitly, 2016), as discussed in Section 3.2 (η = 1.5, τ = 0.5, γ = −0.6 for attention model and η = 1.0, τ = 0.5, γ = −0.1 for MTL), since it was difficult to eliminate the irregular alignments during decoding by only tuning the maximum and minimum lengths and length penalty (we set the minimum and maximum lengths of output sequences by 0.0 and 0.4 times input sequence lengths, respectively and set γ = 0.6 in Table 2).
Table 2 shows the effectiveness of MTL and joint decoding over the attention-based approach, especially showing the significant improvement of the joint CTC-attention decoding. Similar to the CSJ experiments in Section 4.1, we did not use the length-penalty term or the coverage term in joint decoding. This is an advantage of joint decoding over conventional approaches that require many tuning parameters. We also generated more training data by linearly scaling the audio lengths by factors of 0.9 and 1.1 (speed perturb.). The final model achieved 31.4%without using linguistic resources, which defeats moderate state-of-the-art systems including CTC-based methods2.
2Although the proposed method did not reach the performance obtained by time delayed neural network (TDNN) with the lattice-free sequence discriminative training, this method fully utilizes linguistic knowledge through phoneme representation in the discriminative training (Povey et al., 2016), and it is hard to beat it without such knowledge.
5 Summary and discussion
This paper proposes end-to-end ASR by using joint CTC-attention, which outperformed attention-based end-to-end ASR by solving the misalignment issues. This method does not require the use of use linguistic resources including morphological analyzer, pronunciation dictionary, and language model, which are essential component of building conventional Japanese and Mandarin Chinese ASR systems. Nevertheless, the method achieved comparable performance to the state-of-the-art conventional systems for the CSJ and MTS tasks. In addition, the proposed method does not require GMM-HMM construction for initial alignments, DNN pre-training, lattice generation for sequence discriminative training, complex search in decoding (e.g., FST decoder or lexical tree search based decoder). Thus, the method greatly simplifies ASR building process with even smaller amounts of coding. Currently, it took 7- 9 days by using a single GPU to train the network with full training data (581h) in the CSJ task, which is comparable to the whole training time of the conventional state-of-the-art system due to the simplification of building process.
Future work will apply this technique to the other languages including English, where we have to solve an issue of long sequence lengths, which requires heavy computation cost and makes it difficult to train a decoder network. Actually, neural machine translation handles this issue by using a sub word unit (concatenating several letters to form a new sub word unit) (Wu et al., 2016), which would be a promising direction for end-toend ASR.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
A Examples of irregular alignments
We list examples of irregular alignments caused by attention-based ASR. Figure 3 shows an example of repetitions of word chunks. The first chunk of blue characters in attention-based ASR is appeared again, and the whole second chunk part becomes insertion errors. Figure 4 shows an example of deletion errors. The latter half of the sentence in attention-based ASR is broken, which causes deletion errors. The proposed joint CTC-attention avoids these issues.
B Location-based attention mechanism
This section provides the equations of a locationbased attention mechanism Attention(·) in Eq. (10).
alt = Attention({al−1}t,ql−1,ht),
where {al−1}t = [al−1,1, · · · , al−1,T ]>. To obtain alt, we use the following equations:
{flt}t = K ∗ al−1 (21) elt = g
>tanh(Gqql−1 + Ghht + Gfflt + b) (22)
alt = exp(etl)∑ t exp(etl)
(23)
K, Gq, Gh, Gf are matrix parameters. b and g are vector parameters. ∗ denotes convolution along input frame axis t with K.

This paper proposes joint CTC-attention end-to-end ASR, which utilizes both advantages in training and decoding.   - Strengths: It provides a solid work of hybrid CTC-attention framework in training and decoding, and the experimental results showed that the proposed method could provide an improvement in Japanese CSJ and Mandarin Chinese telephone speech recognition task.   - Weaknesses: The only problem is that the paper sounds too similar with Ref [Kim et al., 2016] which will be officially published in the coming IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), March 2017. Kim at al., 2016, proposes joint CTC-attention using MTL for English ASR task, and this paper proposes joint CTC-attention using MTL+joint decoding for Japanese and Chinese ASR tasks. I guess the difference is on joint decoding and the application to Japanese Chinese ASR tasks. However, the difference is not clearly explained by the authors. So it took sometimes to figure out the original contribution of this paper.  (a) Title:  The title in Ref [Kim et al., 2016] is  Joint CTC- Attention Based End-to-End Speech Recognition Using Multi-task Learning d, while the title of this paper is  Joint CTC-attention End-to-end Speech Recognition d. I think the title is too general. If this is the first paper about  Joint CTC-attention  than it is absolutely OK. Or if Ref [Kim et al., 2016] will remain only as pre-published arXiv, then it might be still acceptable. But since [Kim et al., 2016] will officially publish in IEEE conference, much earlier than this paper, then a more specified title that represents the main contribution of this paper in contrast with the existing publication would be necessary.   (b) Introduction: The author claims that  We propose to take advantage of the constrained CTC alignment in a hybrid CTC-attention based system. During training, we attach a CTC objective to an attention-based encoder network as a regularization, as proposed by [Kim at al., 2016].  Taking advantage of the constrained CTC alignment in a hybrid CTC-attention is the original idea from [Kim at al., 2016]. So the whole argument about attention-based end-to-end ASR versus CTC-based ASR, and the necessary of CTC-attention combination is not novel. Furthermore, the statement  we propose  u2026 as proposed by [Kim et al, 2016] d is somewhat weird. We can build upon someone proposal with additional extensions, but not just re-propose other people's proposal. Therefore, what would be important here is to state clearly the original contribution of this paper and the position of the proposed method with respect to existing literature  (c) Experimental Results: Kim at al., 2016 applied the proposed method on English task, while this paper applied the proposed method on Japanese and Mandarin Chinese tasks. I think it would be interesting if the paper could explain in more details about the specific problems in Japanese and Mandarin Chinese tasks that may not appear in English task. For example, how the system could address multiple possible outputs. i.e., Kanji, Hiragana, and Katakana given Japanese speech input without using any linguistic resources. This could be one of the important contributions from this paper.  - General Discussion: I think it would be better to cite Ref [Kim et al., 2016] from the official IEEE ICASSP conference, rather than pre-published arXiv: Kim, S., Hori, T., Watanabe, S.,  Joint CTC- Attention Based End-to-End Speech Recognition Using Multi-task Learning , IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), March 2017, pp. to appear.