Gated ConvNets for Letter-Based ASR
In this paper we introduce a new speech recognition system, leveraging a simple letter-based ConvNet acoustic model. The acoustic model requires only audio transcription for training -- no alignment annotations, nor any forced alignment step is needed. At inference, our decoder takes only a word list and a language model, and is fed with letter scores from the acoustic model -- no phonetic word lexicon is needed. Key ingredients for the acoustic model are Gated Linear Units and high dropout. We show near state-of-the-art results in word error rate on the LibriSpeech corpus with MFSC features, both on the clean and other configurations.
 The paper is interesting, but needs more work, and should provide clear and fair comparisons. Per se, the model is incrementally new, but it is not clear what the strengths are, and the presentations needs to be done more carefully. In detail: - please fix several typos throughout the manuscript, and have a native speaker (and preferably an ASR expert) proofread the paper Introduction - please define HMM/GMM model (and other abbreviations that will be introduced later), it cannot be assumed that the reader is familiar with all of them ("ASG" is used before it is defined, ...) - The standard units that most ASR systems use can be called "senones", and they are context dependent sub-phonetic units (see http://ssli.ee.washington.edu/~mhwang/), not phonetic states. Also the units that generate the alignment and the units that are trained on an alignment can be different (I can use a system with 10000 states to write alignments for a system with 3000 states) - this needs to be corrected. - When introducing CNNs, please also cite Waibel and TDNNs - they are *the same* as 1-d CNNs, and predate them. They have been extended to 2-d later on (Spatio-temporal TDNNs) - The most influential deep learning paper here might be Seide, Li, Yu Interspeech 2011 on CD-DNN-HMMs, rather than overview articles - Many papers get rid of the HMM pipeline, I would add https://arxiv.org/abs/1408.2873, which predates Deep Speech - What is a "sequence-level variant of CTC"? CTC is a sequence training criterion - The reason that Deep Speech 2 is better on noisy test sets is not only the fact they trained on more data, but they also trained on "noisy" (matched) data - how is this an end-to-end approach if you are using an n-gram language model for decoding? Architecture - MFSC are log Filterbanks ... - 1D CNNs would be TDNNs - Figure 2: can you plot the various transition types (normalized, un-normalized, ...) in the plots? not sure if it would help, but it might - Maybe provide a reference for HMM/GMM and EM (forward backward training) - MMI was also widely used in HMM/GMM systems, not just NN systems - the "blank" states do *not* model "garbage" frames, if one wants to interpret them, they might be said to model "non-stationary" frames between CTC "peaks", but these are different from silence, garbage, noise, ... - what is the relationship of the presented ASG criterion to MMI? the form of equation (3) looks like an MMI criterion to me? Experiments - Many of the previous comments still hold, please proofread - you say there is no "complexity" incrase when using "logadd" - how do you measure this? number of operations? is there an implementation of "logadd" that is (absolutely) as fast as "add"? - There is discussion as to what i-vectors model (speaker or environment information) - I would leave out this discussion entirely here, it is enough to mention that other systems use adaptation, and maybe re-run an unadapted baselien for comparsion - There are techniques for incremental adaptation and a constrained MLLR (feature adaptation) approaches that are very eficient, if one wnats to get into this - it may also be interesting to discuss the role of the language model to see which factors influence system performance - some of the other papers might use data augmentation, which would increase noise robustness (did not check, but this might explain some of the results in table 4) - I am confused by the references in the caption of Table 3 - surely the Waibel reference is meant to be for TDNNs (and should appear earlier in the paper), while p-norm came later (Povey used it first for ASR, I think) and is related to Maxout - can you also compare the training times? Conculsion - can you show how your approach is not so computationally expensive as RNN based approaches? either in terms of FLOPS or measured times
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Gated ConvNets for Letter-Based ASR
In this paper we introduce a new speech recognition system, leveraging a simple letter-based ConvNet acoustic model. The acoustic model requires only audio transcription for training -- no alignment annotations, nor any forced alignment step is needed. At inference, our decoder takes only a word list and a language model, and is fed with letter scores from the acoustic model -- no phonetic word lexicon is needed. Key ingredients for the acoustic model are Gated Linear Units and high dropout. We show near state-of-the-art results in word error rate on the LibriSpeech corpus with MFSC features, both on the clean and other configurations.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper is interesting, but needs more work, and should provide clear and fair comparisons. Per se, the model is incrementally new, but it is not clear what the strengths are, and the presentations needs to be done more carefully. In detail: - please fix several typos throughout the manuscript, and have a native speaker (and preferably an ASR expert) proofread the paper Introduction - please define HMM/GMM model (and other abbreviations that will be introduced later), it cannot be assumed that the reader is familiar with all of them ("ASG" is used before it is defined, ...) - The standard units that most ASR systems use can be called "senones", and they are context dependent sub-phonetic units (see http://ssli.ee.washington.edu/~mhwang/), not phonetic states. Also the units that generate the alignment and the units that are trained on an alignment can be different (I can use a system with 10000 states to write alignments for a system with 3000 states) - this needs to be corrected. - When introducing CNNs, please also cite Waibel and TDNNs - they are *the same* as 1-d CNNs, and predate them. They have been extended to 2-d later on (Spatio-temporal TDNNs) - The most influential deep learning paper here might be Seide, Li, Yu Interspeech 2011 on CD-DNN-HMMs, rather than overview articles - Many papers get rid of the HMM pipeline, I would add https://arxiv.org/abs/1408.2873, which predates Deep Speech - What is a "sequence-level variant of CTC"? CTC is a sequence training criterion - The reason that Deep Speech 2 is better on noisy test sets is not only the fact they trained on more data, but they also trained on "noisy" (matched) data - how is this an end-to-end approach if you are using an n-gram language model for decoding? Architecture - MFSC are log Filterbanks ... - 1D CNNs would be TDNNs - Figure 2: can you plot the various transition types (normalized, un-normalized, ...) in the plots? not sure if it would help, but it might - Maybe provide a reference for HMM/GMM and EM (forward backward training) - MMI was also widely used in HMM/GMM systems, not just NN systems - the "blank" states do *not* model "garbage" frames, if one wants to interpret them, they might be said to model "non-stationary" frames between CTC "peaks", but these are different from silence, garbage, noise, ... - what is the relationship of the presented ASG criterion to MMI? the form of equation (3) looks like an MMI criterion to me? Experiments - Many of the previous comments still hold, please proofread - you say there is no "complexity" incrase when using "logadd" - how do you measure this? number of operations? is there an implementation of "logadd" that is (absolutely) as fast as "add"? - There is discussion as to what i-vectors model (speaker or environment information) - I would leave out this discussion entirely here, it is enough to mention that other systems use adaptation, and maybe re-run an unadapted baselien for comparsion - There are techniques for incremental adaptation and a constrained MLLR (feature adaptation) approaches that are very eficient, if one wnats to get into this - it may also be interesting to discuss the role of the language model to see which factors influence system performance - some of the other papers might use data augmentation, which would increase noise robustness (did not check, but this might explain some of the results in table 4) - I am confused by the references in the caption of Table 3 - surely the Waibel reference is meant to be for TDNNs (and should appear earlier in the paper), while p-norm came later (Povey used it first for ASR, I think) and is related to Maxout - can you also compare the training times? Conculsion - can you show how your approach is not so computationally expensive as RNN based approaches? either in terms of FLOPS or measured times
assistant
 The paper is interesting, but needs more work, and should provide clear and fair comparisons. Per se, the model is incrementally new, but it is not clear what the strengths are, and the presentations needs to be done more carefully. In detail: - please fix several typos throughout the manuscript, and have a native speaker (and preferably an ASR expert) proofread the paper Introduction - please define HMM/GMM model (and other abbreviations that will be introduced later), it cannot be assumed that the reader is familiar with all of them ("ASG" is used before it is defined, ...) - The standard units that most ASR systems use can be called "senones", and they are context dependent sub-phonetic units (see http://ssli.ee.washington.edu/~mhwang/), not phonetic states. Also the units that generate the alignment and the units that are trained on an alignment can be different (I can use a system with 10000 states to write alignments for a system with 3000 states) - this needs to be corrected. - When introducing CNNs, please also cite Waibel and TDNNs - they are *the same* as 1-d CNNs, and predate them. They have been extended to 2-d later on (Spatio-temporal TDNNs) - The most influential deep learning paper here might be Seide, Li, Yu Interspeech 2011 on CD-DNN-HMMs, rather than overview articles - Many papers get rid of the HMM pipeline, I would add https://arxiv.org/abs/1408.2873, which predates Deep Speech - What is a "sequence-level variant of CTC"? CTC is a sequence training criterion - The reason that Deep Speech 2 is better on noisy test sets is not only the fact they trained on more data, but they also trained on "noisy" (matched) data - how is this an end-to-end approach if you are using an n-gram language model for decoding? Architecture - MFSC are log Filterbanks ... - 1D CNNs would be TDNNs - Figure 2: can you plot the various transition types (normalized, un-normalized, ...) in the plots? not sure if it would help, but it might - Maybe provide a reference for HMM/GMM and EM (forward backward training) - MMI was also widely used in HMM/GMM systems, not just NN systems - the "blank" states do *not* model "garbage" frames, if one wants to interpret them, they might be said to model "non-stationary" frames between CTC "peaks", but these are different from silence, garbage, noise, ... - what is the relationship of the presented ASG criterion to MMI? the form of equation (3) looks like an MMI criterion to me? Experiments - Many of the previous comments still hold, please proofread - you say there is no "complexity" incrase when using "logadd" - how do you measure this? number of operations? is there an implementation of "logadd" that is (absolutely) as fast as "add"? - There is discussion as to what i-vectors model (speaker or environment information) - I would leave out this discussion entirely here, it is enough to mention that other systems use adaptation, and maybe re-run an unadapted baselien for comparsion - There are techniques for incremental adaptation and a constrained MLLR (feature adaptation) approaches that are very eficient, if one wnats to get into this - it may also be interesting to discuss the role of the language model to see which factors influence system performance - some of the other papers might use data augmentation, which would increase noise robustness (did not check, but this might explain some of the results in table 4) - I am confused by the references in the caption of Table 3 - surely the Waibel reference is meant to be for TDNNs (and should appear earlier in the paper), while p-norm came later (Povey used it first for ASR, I think) and is related to Maxout - can you also compare the training times? Conculsion - can you show how your approach is not so computationally expensive as RNN based approaches? either in terms of FLOPS or measured times