End-to-end automatic speech recognition (ASR) has become a popular alternative to conventional DNN/HMM hybrid systems because it avoids the need for linguistic resources such as pronunciation dictionary, tokenization, and contextdependency trees, leading to a greatly simplified model-building process. There are two major types of end-to-end architectures for ASR: attention-based methods use an attention mechanism to perform alignment between acoustic frames and recognized symbols, and connectionist temporal classification (CTC), uses Markov assumptions to efficiently solve sequential problems by dynamic programming. This paper proposes joint CTCattention end-to-end ASR, which effectively utilizes both advantages in training and decoding. We have applied the proposed method to two ASR benchmarks (spontaneous Japanese and Mandarin Chinese), and showing the comparable performance to conventional hybrid ASR systems without linguistic resources.
1 Introduction
Automatic speech recognition (ASR) is currently a mature set of technologies that have been widely deployed, resulting in great success in interface applications such as voice search. A typical ASR system is factorized into several modules including acoustic, lexicon, and language models based on a probabilistic noisy channel model (Jelinek, 1976). Over the last decade, dramatic improvements in acoustic and language models have been driven by machine learning techniques known as deep learning (Hinton et al., 2012). However,
current systems lean heavily on the scaffolding of complicated legacy architectures that grew up around traditional techniques. These present the following problems that we may seek to eliminate.
1. Flat start: many module-specific processes are required to build an accurate module: for example, when we build an acoustic model from scratch, we have to first build hidden Markov model (HMM) and Gaussian mixture model (GMM) followed by deep neural networks (DNN). 2. Linguistic knowledge: to well factorize acoustic and language models, we need to have a lexicon model, which is usually based on a hand-crafted pronunciation dictionary to map word to phoneme sequence. Also, some languages do not explicitly have a word boundary and need tokenization modules (Kudo et al., 2004; Bird, 2006). 3. Conditional independence assumptions: the current ASR systems often use conditional independence assumptions (especially Markov assumptions) during the above factorization and to make use of GMM, DNN, and n-gram models. The data do not necessarily follow such assumptions leading to model mis-specification. 4. Complex decoding: inference/decoding has to be performed by integrating all modules. Although this integration is often efficiently handled by finite state transducers, the construction and implementation of welloptimized transducers is very complicated. 5. Local optimum: the above modules are optimized separately, which may result in local optima, where each module is trained to match the other modules1.
1 Sequence discriminative training (Veselỳ et al., 2013) can solve the issue to some extent, but requires additional
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Consequently, it is quite difficult for non-experts to use/develop ASR systems for new applications, especially for new languages.
End-to-end ASR has the goal of simplifying the above module-based architecture into a singlenetwork architecture within a deep learning framework, in order to address the above issues. There are two major types of end-to-end architectures for ASR: attention-based methods use an attention mechanism to perform alignment between acoustic frames and recognized symbols, and connectionist temporal classification (CTC), uses Markov assumptions to efficiently solve sequential problems by dynamic programming (Chorowski et al., 2014; Graves and Jaitly, 2014).
All ASR models aim to elucidate the posterior distribution p(W |X) of word sequence W given speech feature sequence X . The end-to-end methods do this directly whereas conventional models factorize p(W |X) into modules such as the language model p(W ), which can be trained on pure language data, and an acoustic model likelihood p(X|W ) which is trained on acoustic data with the corresponding language labels. End-to-end ASR methods typically rely only on paired acoustic and language data. Without the additional language data, they can suffer from data sparseness or outof-vocabulary issues. To improve generalization, and handle out-of-vocabulary problems, it is typical to use the letter representation rather than the word representation for the language output sequence, which we adopt in the descriptions below.
The attention-based end-to-end method solves the ASR problem as a sequence mapping from speech feature sequences to text by using encoderdecoder architecture. The decoder network uses an attention mechanism to find an alignment between each element of the output sequence and the hidden states generated by the acoustic encoder network for each frame of acoustic input (Chorowski et al., 2014, 2015; Chan et al., 2015; Lu et al., 2016). At each output position, the decoder network computes a matching score between its hidden state and the states of the encoder network at each input time, to form a temporal alignment distribution, which is then used to extract an average of the corresponding encoder states.
This basic temporal attention mechanism is too flexible in the sense that it allows extremely non-
process in the above modules including lattice generations.
sequential alignments. This may be fine for applications such as machine translation where input and output word order are different (Bahdanau et al., 2014; Wu et al., 2016). However in speech recognition, the feature inputs and corresponding letter outputs generally proceed in the same order, with only small within-word deviations (e.g. the word ”iron” which transposes the sounds for ”r” and ”o”). Another problem is that the input and output sequences in ASR can have very different lengths, and these vary greatly from case to case, depending on the speaking rate and writing system, making it more difficult to track the alignment.
However, an advantage is that the attention mechanism does not require any conditional independence assumptions, and could address all the problems cited above. Although the alignment problems of attention-based mechanisms have been partially addressed in (Chorowski et al., 2014; Chorowski and Jaitly, 2016) using various mechanisms, here we propose more rigorous constraints by using CTC-based alignment to guide the training.
CTC permits an efficient computation of a strictly monotonic alignment using dynamic programming (Graves et al., 2006; Graves and Jaitly, 2014) although it requires language models and graph-based decoding (Miao et al., 2015) except in the case of huge training data (Amodei et al., 2015; Soltau et al., 2016). We propose to take advantage of the constrained CTC alignment in a hybrid CTC-attention based system. During training, we attach a CTC objective to an attention-based encoder network as a regularization, as proposed by (Kim et al., 2016). This greatly reduces irregular alignments without any heuristic search techniques. During decoding, we propose to use a rescoring technique, where hypotheses of attentionbased ASR are refined by scores obtained by using encoder outputs.
The proposed method is applied to Japanese and Mandarin ASR tasks, which require extra linguistic resources including morphological analyzer (Kudo et al., 2004) or word segmentation (Xue et al., 2003) in addition to pronunciation dictionary to provide accurate lexicon and language models in conventional hybrid ASR. Surprisingly, the method achieved performance comparable to, and in some cases superior to, several state-of-theart hybrid ASR systems, without using the above
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
linguistic resources.
2 From hybrid to end-to-end ASR
This section briefly provides a formulation of conventional HMM/DNN hybrid ASR and CTC or attention based end-to-end ASR. The formulation is intended to clarify the conditional independence assumption (Markov assumption), which is an important property to characterize these three methods.
2.1 Hybrid HMM/DNN
ASR deals with a sequence mapping from T - length speech feature sequence X = {xt ∈ RD|t = 1, · · · , T} to N -length word sequence W = {wn ∈ V|n = 1, · · · , N}. xt is a D dimensional speech feature vector (e.g., log Mel filterbanks) at frame t and wn is a word at position n in vocabulary V .
ASR is mathematically formulated with the Bayes decision theory, where the most probable word sequence Ŵ is estimated among all possible word sequences V∗ as follows:
Ŵ = arg max W∈V∗
p(W |X). (1)
Therefore, the main problem of ASR is how to obtain the posterior distribution p(W |X).
In the current main stream of ASR is called hybrid HMM/DNN (Bourlard and Morgan, 1994), which uses the Bayes theorem and introduces HMM state sequence S = {st ∈ {1, · · · , J}|t = 1, · · · , T} to factorize p(W |X) into the following three distributions:
arg max W
p(W |X)
≈ arg max W ∑ S p(X|S)p(S|W )p(W ). (2)
The three factors, p(X|S), p(S|W ), and p(W ), are acoustic, lexicon, and language models, respectively. Eq. (2) is obtained by a conditional independence assumption (i.e., p(X|S,W ) ≈ p(X|S)), which is a reasonable assumption to simplify the dependency of the acoustic model.
Acoustic model p(X|S) p(X|S) is further factorized by using a probabilistic chain rule and conditional independence assumption as follows:
p(X|S) ≈ ∏ t p(xt|st) ∝ ∏ t p(st|xt) p(st) , (3)
where the framewise likelihood function p(xt|st) is replaced with the framewise posterior distribution p(st|xt)/p(st) computed by powerful DNN classifiers by using so-called pseudo likelihood trick (Bourlard and Morgan, 1994). The conditional independence assumption in Eq. (3) is often regarded as too strong assumption, since it does not consider any input and hidden state contexts. Therefore DNNs with long context features or recurrent neural networks are often used to mitigate the issue. To train the framewise posterior, we also require to provide a framewise state alignment st as a target, which is often provided by a HMM/GMM system.
Lexicon model p(S|W ) p(S|W ) is also factorized by using a probabilistic chain rule and conditional independence assumption (1st-order Markov assumption) as follows:
p(S|W ) ≈ ∏ t p(st|st−1,W ) (4)
This probability is represented by an HMM state transition given W . The conversion from W to HMM states is deterministically performed by using a pronunciation dictionary through a phoneme representation.
Language model p(W ) Similarly, p(W ) is factorized by using a probabilistic chain rule and conditional independence assumption (m − 1th-order Markov assumption) as a m-gram model, i.e., p(W ) ≈∏
n p(wn|wn−1, . . . , wn−m−1). Although recurrent neural network language models (RNNLMs) can avoid this conditional independence assumption issue (Mikolov et al., 2010), it makes the decoding complex, and RNNLMs are often combined with m-gram language models based on as a re-scoring technique.
Thus, conventional hybrid HMM/DNN systems make the ASR problem formulated with Eq. (1) feasible by using factorization and conditional independence assumptions, at the cost of the five problems discussed in Section 1.
2.2 Connectionist Temporal Classification (CTC)
The CTC formulation also follows from Bayes decision theory (Eq. (1)). Note that the CTC formulation uses L-length letter sequence C = {cl ∈ U|l = 1, · · · , L} with a set of distinct letters U .
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Similarly to Section 2.1, by introducing framewise letter sequence with an additional ”blank” symbol Z = {zt ∈ U ∪ blank|t = 1, · · · , T}, and by using the probabilistic chain rule and conditional independence assumption, the posterior distribution p(C|X) is factorized as follows:
p(C|X) ≈ ∑ Z p(C|Z)p(Z|X) ≈ ∑ Z p(C|Z) ∏ t p(zt|X)
≈ ∑ Z ∏ t
p(zt|zt−1, C)p(zt|X)︸ ︷︷ ︸ ,pctc(C|X) p(C) (5)
CTC is similar to the hybrid approach, except that it applies Bayes theorem to p(C|Z) instead of to p(W |X). As a result, CTC has three distribution components similar to the hybrid case, i.e., framewise posterior distribution p(zt|X), transition probability p(zt|zt−1, C), and letter-based language model p(C). We also define the CTC objective function pctc(C|X) used in the later formulation.
The framewise posterior distribution p(zt|X) is conditioned on all inputs X , and it is quite natural to be modeled by using bidirectional long shortterm memory (BLSTM):
p(zt|X) = Softmax(Lin(ht)) (6) ht = BLSTM(X). (7)
Softmax(·) is a sofmax activation function, and Lin(·) is a linear layer to convert hidden vector ht to a (|U| + 1) dimensional vector (+1 means a blank symbol introduced in CTC).
Although Eq. (5) has to deal with a summation over all possible Z, it is efficiently computed by using dynamic programming (Viterbi/forwardbackward algorithm) thanks to the Markov property. In summary, although CTC and hybrid systems are similar to each other due to conditional independence assumptions, CTC does not require pronunciation dictionaries and omits an HMM/GMM construction step.
2.3 Attention mechanism
Compared with hybrid and CTC approaches, the attention-based approach does not make any conditional independence assumptions, and directly estimates the posterior p(C|X) based on a prob-
abilistic chain rule, as follows: p(C|X) = ∏ l
p(cl|c1, · · · , cl−1, X)︸ ︷︷ ︸ ,patt(C|X) , (8)
where patt(C|X) is an attention-based objective function. p(cl|c1, · · · , cl−1, X) is obtained by
p(cl|c1, · · · , cl−1, X) = Decoder(rl,ql−1, cl−1) ht = Encoder(X) (9)
alt = Attention({al−1}t,ql−1,ht) (10) rl = ∑ t altht. (11)
Eq. (9) converts input feature vectors X into a framewise hidden vector ht in an encoder network based on BLSTM, i.e., Encoder(X) , BLSTM(X). Attention(·) in Eq. (10) is based on a content-based attention mechanism with convolutional features, as described in (Chorowski et al., 2015) (see Appendix B). alt is an attention weight, and represents a soft alignment of hidden vector ht for each output cl based on the weighted summation of hidden vectors to form letter-wise hidden vector rl in Eq. (11). A decoder network is another recurrent network conditioned on previous output cl−1 and hidden vector ql−1, similar to RNNLM, in addition to letter-wise hidden vector rl. We use Decoder(·) , Softmax(Lin(LSTM(·))).
Attention-based ASR does not explicitly separate each module, and potentially handles the all five issues pointed out in Section 1. It implicitly combines acoustic models, lexicon, and language models as encoder, attention, and decoder networks, which can be jointly trained as a single deep neural network.
Compared with hybrid and CTC, which are based on a transition form from t − 1 to t due to the Markov assumption, the attention mechanism does not maintain this constraint, and often provides irregular alignments. A major focus of this paper is to address this problem by proposing joint CTC-attention models.
3 Joint CTC-attention
This section explains a joint CTC-attention network, which utilizes both benefits of CTC and attention during training and decoding steps in ASR.
3.1 Multi-task learning
Kim et al. (2016) uses a CTC objective function as an auxiliary task to train the attention model en-
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
ㅡ ㅡㅡz2 …
sos eosc1
q0
r0 r1 rL
H
h2 h4
q1 qL
hT
x1 x2 x3 x4 x5 x6 xT
Shared Encoder
CTC
Attention Decoder
x7 x8
z4
h6 h8
…
c2
r2
q2
…c1 c2 …
c1 c2 …
Figure 1: Joint CTC-attention based end-to-end framework: the shared encoder is trained by both CTC and attention model objectives simultaneously. The shared encoder transforms our input sequence {xt · · ·xT } into high level featuresH = {ht · · ·hT }, and the attention decoder generates the letter sequence {c1 · · · cL}.
coder within the multi-task learning (MTL) framework. Figure 1 illustrates the overall architecture of the framework, where the same BLSTM is shared with CTC and attention encoder networks (that is Eqs. (7) and (9), respectively). Unlike the sole attention model, the forward-backward algorithm of CTC can enforce monotonic alignment between speech and label sequences during training. That is, rather than solely depending on data-driven attention methods to estimate the desired alignments in long sequences, the forwardbackward algorithm in CTC helps to speed up the process of estimating the desired alignment. The objective to be maximized is a logarithmic linear combination of the CTC and attention objectives, i.e., pctc(C|X) in Eq. (5) and patt(C|X) in Eq. (8):
LMTL = λ log pctc(C|X) + (1− λ) log patt(C|X), (12)
with a tunable parameter λ : 0 ≤ λ ≤ 1.
3.2 Joint CTC-attention decoding
The inference step of our joint CTC-attention based end-to-end speech recognition is performed by output-label synchronous decoding with a beam search like attention-based ASR. But, we take the CTC probabilities into account to find a better aligned hypothesis to the input speech, as shown in Figure 1
End-to-end speech recognition inference is generally defined as a problem to find the most probable letter sequence Ĉ given speech input X , i.e.
Ĉ = arg max C∈U∗
log p(C|X). (13)
In attention-based ASR, p(C|X) is computed by Eq. (8), and Ĉ is found by the beam search.
Let Ωl be a set of partial hypotheses of length l. At the beginning of the beam search, Ω0 contains only one hypothesis with starting symbol <sos>. For l = 1 to Lmax, each partial hypothesis in Ωl−1 is expanded by appending possible single letters, and the new hypotheses are stored in Ωl, where Lmax is the maximum length of hypotheses to be searched. The score of each new hypothesis is computed in log domain as
α(gl−1 · c) = α(gl−1) + log p(c|gl−1, X), (14)
where gl−1 is a partial hypothesis in Ωl−1 and c is a letter appended to gl−1. The new partial hypothesis gl, i.e. gl−1 · c, is added to Ωl. If c is a special symbol that represents the end of sequence, <eos>, gl is added to Ω̂ instead of Ωl, where Ω̂ denotes a set of complete hypotheses. Finally, Ĉ is obtained as
Ĉ = arg max g∈Ω̂ α(g). (15)
In the beam search, Ωl is allowed to hold only a limited number of hypotheses with higher scores to improve the search efficiency.
Attention-based ASR, however, may be prone to deletion and insertion errors (see Appendix A) because of its flexible alignment property, which can attend any portion of the encoder state sequence to predict the next label, as discussed in Section 2.3. Since the attention is generated by the decoder network it may prematurely predict the end-of-sequence label, even when it has not attended all the encoder frames, making the hypothesis too short. On the other hand, it may predict the next label with a high probability by attending the same portions as those attended before. In this case, the hypothesis becomes very long and includes repetitions of the same label sequence.
To alleviate this problem, a length penalty term is commonly used to control the hypothesis length to be selected (Chorowski et al., 2015; Bahdanau et al., 2016). With the length penalty, the decoding objective is changed to
Ĉ = arg max C∈U∗
{log p(C|X) + γ|C|} , (16)
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
where |C| is the length of sequence C and γ is a tunable parameter. However, it is actually difficult to completely exclude too short and too long hypotheses even if γ is carefully tuned. It is also effective to control the hypothesis length by minimum and maximum lengths to some extent, where the minimum and the maximum are selected as fixed rations to the length of input speech. But, since there are exceptionally long or short transcripts compared to the input speech, it is difficult to balance saving such exceptional transcripts and preventing hypotheses with irrelevant lengths.
Another approach is a coverage term recently proposed in (Chorowski and Jaitly, 2016), which is incorporated in the decoding objective as
Ĉ = arg max C∈U∗
{log p(C|X) + γ|C|
+η · coverage(C|X)} , (17)
and computed by
coverage(C|X) = T∑ t=1 [ L∑ l=1 alt > τ ] , (18)
where η and τ are tunable parameters. The coverage term represents the number of frames that have received a cumulative attention greater than τ . Accordingly, it increases when paying close attention to some frames for the first time, but does not increase when paying attention again to the same frames. This property is effective to avoid looping the same label sequence within a hypothesis. However, it is still difficult to obtain a common parameter setting for γ, η, τ and optional min/max lengths so that they are appropriate for any speech data from different tasks.
Our joint CTC-attention approach combines CTC and attention-based sequence probabilities in the inference step, as well as the training step. Suppose pctc(C|X) in Eq. (5) and patt(C|X) in Eq. (8) are the sequence probabilities given by CTC and the attention model, respectively. The decoding objective is defined similarly to Eq. (12) as
Ĉ = arg max C∈U∗
{λ log pctc(C|X)
+(1− λ) log patt(C|X)} . (19)
The CTC probability enforces a monotonic alignment that does not allow big jumps or looping the same frames. Accordingly, it is possible to
choose a hypothesis with a better alignment and exclude irrelevant hypotheses without relying on any of the coverage term, the length penalty, and the min/max lengths.
We implement the joint CTC-attention inference method as a two-pass strategy. The first pass obtains a set of complete hypotheses using the beam search, where only attention-based sequence probabilities are considered. The second pass rescores the complete hypotheses using the CTC-attention probabilities, where the CTC probabilities are obtained by the forward algorithm for CTC (Graves et al., 2006). The rescoring pass obtains the final result according to
Ĉ = arg max g∈Ω̂ {(1− λ)α(g) + λξ(g)} , (20)
where ξ(g) = log pctc(g|X).
4 Experiments
We used Japanese and Mandarin Chinese ASR benchmarks to show the effectiveness of the proposed joint CTC-attention approach. The main reason for choosing these two languages is that those ideogram languages have relatively shorter lengths for letter sequences than those in alphabet languages, which reduces computational complexities greatly, and makes it easy to handle context information in a decoder network. Our preliminary investigation shows that Japanese and Mandarin Chinese end-to-end ASR can be easily scaled up, and shows state-of-the-art performance without using various tricks developed in English tasks.
4.1 Corpus of Spontaneous Japanese (CSJ)
We demonstrated ASR experiments by using the Corpus of Spontaneous Japanese (CSJ) (Maekawa et al., 2000). CSJ is a standard Japanese ASR task based on a collection of monologue speech data including academic lectures and simulated presentations. It has a total of 581 hours of training data and three types of evaluation data, where each evaluation task consists of 10 lectures (totally 5 hours). As input features, we used 40 melscale filterbank coefficients, with their first and second order temporal derivatives to obtain a total of 120-dimensional feature vector per frame. The encoder was a 4-layer BLSTM with 320 cells in each layer and direction, and linear projection layer is followed by each BLSTM layer. The
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Table 1: Character error rate (CER) for conventional attention and proposed joint CTC-attention end-toend ASR. Corpus of Spontaneous Japanese speech recognition (CSJ) task.
Model Hour Task1 Task2 Task3 Attention 581 11.4 7.9 9.0 MTL 581 10.5 7.6 8.3 MTL + joint decoding 581 10.1 7.1 7.8 MTL-large + joint decoding 581 9.4 7.0 7.5 GMM-discr. (Moriya et al., 2015) 236 for AM, 581 for LM 11.2 9.2 12.1 DNN-hybrid (Moriya et al., 2015) 236 for AM, 581 for LM 9.0 7.2 9.6 CTC-syllable (Kanda et al., 2016) 581 9.4 7.3 7.5
2nd and 3rd bottom layers of the encoder read every second hidden state in the network below, reducing the utterance length by the factor of 4. We used the content-based attention mechanism (Chorowski et al., 2015), where the 10 centered convolution filters of width 100 were used to extract the convolutional features. The decoder network was a 1-layer LSTM with 320 cells. The AdaDelta algorithm (Zeiler, 2012) with gradient clipping (Pascanu et al., 2012) was used for the optimization. The joint CTC-attention ASR was implemented by using the Chainer deep learning toolkit (Tokui et al., 2015).
Table 1 first compares the character error rate (CER) for conventional attention and MTL based end-to-end ASR without the joint decoding. λ in Eq. (12) was set to 0.1. When decoding, we manually set the minimum and maximum lengths of output sequences by 0.1 and 0.5 times input sequence lengths, respectively. The length penalty γ in Eq. (16) was set to 0.1. MTL significantly outperformed attention-based ASR in the all evaluation tasks, which confirms the effectiveness of joint CTC-attention model. Table 1 also shows that the joint decoding, described in Section 3.2, further improved the performance without setting any search parameters (maximum and minimum lengths, length penalty), but only setting a weight parameter λ = 0.1 in Eq. (20) similar to the MTL case. Figure 2 also compares the dependency of λ on the CER for the CSJ evaluation tasks, and showing that λ was not so sensitive to the performance if we set λ around the value we used at MTL (i.e., 0.1).
We also compare the performance of the proposed MTL-large, which has a larger network (5-layer encoder network), with the conventional state-of-the-art techniques obtained by using linguistic resources. The state-of-the-art CERs of GMM discriminative training and DNN-sMBR
6.0
7.0
8.0
9.0
10.0
11.0
12.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
CE R  (% )
Re‐scoring weight
Task1 Task2 Task3
Figure 2: The effect of re-scoring weight λ in Eq. (20) on the CSJ evaluation tasks.
hybrid systems are obtained from the Kaldi recipe (Moriya et al., 2015) and a system based on syllable-based CTC with MAP decoding (Kanda et al., 2016). The Kaldi recipe systems use academic lectures (236h) for AM training and all training-data transcriptions for LM training. Unlike the proposed method, these methods use linguistic resources including a morphological analyzer, pronunciation dictionary, and language model. Note that since the amount of training data and experimental configurations of the proposed and reference methods are different, it is difficult to compare the performance listed in the table directly. However, since the CERs of the proposed method are comparable to those of the best reference results, we can state that the proposed method reaches the state-of-the-art performance.
4.2 Mandarin telephone speech
We demonstrated ASR experiments on HKUST Mandarin Chinese conversational telephone speech recognition (MTS) (Liu et al., 2006). It has 5 hours recording for evaluation, and we extracted 5 hours from training data as a development set, and used the rest (167 hours) as a training set. All experimental conditions were same as those in Section 4.1 except that we used the λ = 0.5 in training and decoding instead of
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Table 2: Character error rate (CER) for conventional attention and proposed joint CTC-attention end-toend ASR. HKUST Mandarin Chinese conversational telephone speech recognition (MTS) task.
Model dev eval Attention 40.3 37.8 MTL 38.7 36.6 Attention + coverage 39.4 37.6 MTL + coverage 36.9 35.3 MTL + joint decoding 35.9 34.2 MTL + joint decoding (speed perturb.) 32.1 31.4 DNN-hybrid – 35.9 LSTM-hybrid (speed perturb.) – 33.5 CTC with language model (Miao et al., 2016) – 34.8 TDNN-hybrid, lattice-free MMI (speed perturb.) (Povey et al., 2016) – 28.2
0.1 based on our preliminary investigation and 80 mel-scale filterbank coefficients with pitch feature as an input by referring (Miao et al., 2016). In decoding, we also added a result of the coverage-term based decoding (Chorowski and Jaitly, 2016), as discussed in Section 3.2 (η = 1.5, τ = 0.5, γ = −0.6 for attention model and η = 1.0, τ = 0.5, γ = −0.1 for MTL), since it was difficult to eliminate the irregular alignments during decoding by only tuning the maximum and minimum lengths and length penalty (we set the minimum and maximum lengths of output sequences by 0.0 and 0.4 times input sequence lengths, respectively and set γ = 0.6 in Table 2).
Table 2 shows the effectiveness of MTL and joint decoding over the attention-based approach, especially showing the significant improvement of the joint CTC-attention decoding. Similar to the CSJ experiments in Section 4.1, we did not use the length-penalty term or the coverage term in joint decoding. This is an advantage of joint decoding over conventional approaches that require many tuning parameters. We also generated more training data by linearly scaling the audio lengths by factors of 0.9 and 1.1 (speed perturb.). The final model achieved 31.4%without using linguistic resources, which defeats moderate state-of-the-art systems including CTC-based methods2.
2Although the proposed method did not reach the performance obtained by time delayed neural network (TDNN) with the lattice-free sequence discriminative training, this method fully utilizes linguistic knowledge through phoneme representation in the discriminative training (Povey et al., 2016), and it is hard to beat it without such knowledge.
5 Summary and discussion
This paper proposes end-to-end ASR by using joint CTC-attention, which outperformed attention-based end-to-end ASR by solving the misalignment issues. This method does not require the use of use linguistic resources including morphological analyzer, pronunciation dictionary, and language model, which are essential component of building conventional Japanese and Mandarin Chinese ASR systems. Nevertheless, the method achieved comparable performance to the state-of-the-art conventional systems for the CSJ and MTS tasks. In addition, the proposed method does not require GMM-HMM construction for initial alignments, DNN pre-training, lattice generation for sequence discriminative training, complex search in decoding (e.g., FST decoder or lexical tree search based decoder). Thus, the method greatly simplifies ASR building process with even smaller amounts of coding. Currently, it took 7- 9 days by using a single GPU to train the network with full training data (581h) in the CSJ task, which is comparable to the whole training time of the conventional state-of-the-art system due to the simplification of building process.
Future work will apply this technique to the other languages including English, where we have to solve an issue of long sequence lengths, which requires heavy computation cost and makes it difficult to train a decoder network. Actually, neural machine translation handles this issue by using a sub word unit (concatenating several letters to form a new sub word unit) (Wu et al., 2016), which would be a promising direction for end-toend ASR.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
A Examples of irregular alignments
We list examples of irregular alignments caused by attention-based ASR. Figure 3 shows an example of repetitions of word chunks. The first chunk of blue characters in attention-based ASR is appeared again, and the whole second chunk part becomes insertion errors. Figure 4 shows an example of deletion errors. The latter half of the sentence in attention-based ASR is broken, which causes deletion errors. The proposed joint CTC-attention avoids these issues.
B Location-based attention mechanism
This section provides the equations of a locationbased attention mechanism Attention(·) in Eq. (10).
alt = Attention({al−1}t,ql−1,ht),
where {al−1}t = [al−1,1, · · · , al−1,T ]>. To obtain alt, we use the following equations:
{flt}t = K ∗ al−1 (21) elt = g
>tanh(Gqql−1 + Ghht + Gfflt + b) (22)
alt = exp(etl)∑ t exp(etl)
(23)
K, Gq, Gh, Gf are matrix parameters. b and g are vector parameters. ∗ denotes convolution along input frame axis t with K.
This paper proposes joint CTC-attention end-to-end ASR, which utilizes both advantages in training and decoding.   - Strengths: It provides a solid work of hybrid CTC-attention framework in training and decoding, and the experimental results showed that the proposed method could provide an improvement in Japanese CSJ and Mandarin Chinese telephone speech recognition task.   - Weaknesses: The only problem is that the paper sounds too similar with Ref [Kim et al., 2016] which will be officially published in the coming IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), March 2017. Kim at al., 2016, proposes joint CTC-attention using MTL for English ASR task, and this paper proposes joint CTC-attention using MTL+joint decoding for Japanese and Chinese ASR tasks. I guess the difference is on joint decoding and the application to Japanese Chinese ASR tasks. However, the difference is not clearly explained by the authors. So it took sometimes to figure out the original contribution of this paper.  (a) Title:  The title in Ref [Kim et al., 2016] is  Joint CTC- Attention Based End-to-End Speech Recognition Using Multi-task Learning d, while the title of this paper is  Joint CTC-attention End-to-end Speech Recognition d. I think the title is too general. If this is the first paper about  Joint CTC-attention  than it is absolutely OK. Or if Ref [Kim et al., 2016] will remain only as pre-published arXiv, then it might be still acceptable. But since [Kim et al., 2016] will officially publish in IEEE conference, much earlier than this paper, then a more specified title that represents the main contribution of this paper in contrast with the existing publication would be necessary.   (b) Introduction: The author claims that  We propose to take advantage of the constrained CTC alignment in a hybrid CTC-attention based system. During training, we attach a CTC objective to an attention-based encoder network as a regularization, as proposed by [Kim at al., 2016].  Taking advantage of the constrained CTC alignment in a hybrid CTC-attention is the original idea from [Kim at al., 2016]. So the whole argument about attention-based end-to-end ASR versus CTC-based ASR, and the necessary of CTC-attention combination is not novel. Furthermore, the statement  we propose  u2026 as proposed by [Kim et al, 2016] d is somewhat weird. We can build upon someone proposal with additional extensions, but not just re-propose other people's proposal. Therefore, what would be important here is to state clearly the original contribution of this paper and the position of the proposed method with respect to existing literature  (c) Experimental Results: Kim at al., 2016 applied the proposed method on English task, while this paper applied the proposed method on Japanese and Mandarin Chinese tasks. I think it would be interesting if the paper could explain in more details about the specific problems in Japanese and Mandarin Chinese tasks that may not appear in English task. For example, how the system could address multiple possible outputs. i.e., Kanji, Hiragana, and Katakana given Japanese speech input without using any linguistic resources. This could be one of the important contributions from this paper.  - General Discussion: I think it would be better to cite Ref [Kim et al., 2016] from the official IEEE ICASSP conference, rather than pre-published arXiv: Kim, S., Hori, T., Watanabe, S.,  Joint CTC- Attention Based End-to-End Speech Recognition Using Multi-task Learning , IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), March 2017, pp. to appear.