Latent Sequence Decompositions
Sequence-to-sequence models rely on a fixed decomposition of the target sequences into a sequence of tokens that may be words, word-pieces or characters. The choice of these tokens and the decomposition of the target sequences into a sequence of tokens is often static, and independent of the input, output data domains. This can potentially lead to a sub-optimal choice of token dictionaries, as the decomposition is not informed by the particular problem being solved. In this paper we present Latent Sequence Decompositions (LSD), a framework in which the decomposition of sequences into constituent tokens is learnt during the training of the model. The decomposition depends both on the input sequence and on the output sequence. In LSD, during training, the model samples decompositions incrementally, from left to right by locally sampling between valid extensions. We experiment with the Wall Street Journal speech recognition task. Our LSD model achieves 12.9% WER compared to a character baseline of 14.8% WER. When combined with a convolutional network on the encoder, we achieve a WER of 9.6%.
This submission proposes to learn the word decomposition, or word to sub-word sequence mapping jointly with the attention based sequence-to-sequence model. A particular feature of this approach is that the decomposition is not static, instead, it also conditions on the acoustic input, and the mapping is probabilistic, i.e., one word may map to multiple sub-word sequences. The authors argue that the dynamic decomposition approach can more naturally reflect the acoustic pattern. Interestingly, the motivation behind this approach is analogous to learning the pronunciation mixture model for HMM based speech recognition, where the probabilistic mapping from a word to its pronunciations also conditions on the acoustic input, e.g., I. McGraw, I. Badr, and J. Glass, *Learning lexicons form speech using a pronunciation mixture model,* in IEEE Transactions on Audio, Speech, and Language Processing, 2013 L. Lu, A. Ghoshal, S. Renals, *Acoustic data-driven pronunciation lexicon for large vocabulary speech recognition*, in Proc. ASRU R. Singh, B. Raj, and R. Stern, *Automatic generation of subword units for speech recognition systems,* in IEEE Transactions on Speech and Audio Processing, 2002 It would be interesting to put this work in the context by linking it to some previous works in the HMM framework. Overall, the paper is well written, and it is theoretically convincing. The experimental study could be more solid, e.g., it is reasonable to have a word-level baseline, as the proposed approach lies in between the character-level and word-level systems. the vocabulary size of the WSJ si284 dataset is 20K at maximum, which is not very large for the softmax layer, and it is a closed vocabulary task. I guess the word-level system may be also competitive to the numbers reported in this paper. Furthermore, can you explain what is the computational bottleneck of the proposed approach? You downsampled the data by the factor of 4 using an RNN, and it still took around 5 days to converge. To me, it is a bit expensive, especially given that you only take one sample when computing the gradient. Table 2 is a little bit misleading, as CTC with language model and seq2seq with a language model model from Bahdanau et al. is much closer to the best number reported in this Table 2, while you may only get a very small improvement using a language model. Finally, *O(5) days to converge* sounds a bit odd to me.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Latent Sequence Decompositions
Sequence-to-sequence models rely on a fixed decomposition of the target sequences into a sequence of tokens that may be words, word-pieces or characters. The choice of these tokens and the decomposition of the target sequences into a sequence of tokens is often static, and independent of the input, output data domains. This can potentially lead to a sub-optimal choice of token dictionaries, as the decomposition is not informed by the particular problem being solved. In this paper we present Latent Sequence Decompositions (LSD), a framework in which the decomposition of sequences into constituent tokens is learnt during the training of the model. The decomposition depends both on the input sequence and on the output sequence. In LSD, during training, the model samples decompositions incrementally, from left to right by locally sampling between valid extensions. We experiment with the Wall Street Journal speech recognition task. Our LSD model achieves 12.9% WER compared to a character baseline of 14.8% WER. When combined with a convolutional network on the encoder, we achieve a WER of 9.6%.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This submission proposes to learn the word decomposition, or word to sub-word sequence mapping jointly with the attention based sequence-to-sequence model. A particular feature of this approach is that the decomposition is not static, instead, it also conditions on the acoustic input, and the mapping is probabilistic, i.e., one word may map to multiple sub-word sequences. The authors argue that the dynamic decomposition approach can more naturally reflect the acoustic pattern. Interestingly, the motivation behind this approach is analogous to learning the pronunciation mixture model for HMM based speech recognition, where the probabilistic mapping from a word to its pronunciations also conditions on the acoustic input, e.g., I. McGraw, I. Badr, and J. Glass, *Learning lexicons form speech using a pronunciation mixture model,* in IEEE Transactions on Audio, Speech, and Language Processing, 2013 L. Lu, A. Ghoshal, S. Renals, *Acoustic data-driven pronunciation lexicon for large vocabulary speech recognition*, in Proc. ASRU R. Singh, B. Raj, and R. Stern, *Automatic generation of subword units for speech recognition systems,* in IEEE Transactions on Speech and Audio Processing, 2002 It would be interesting to put this work in the context by linking it to some previous works in the HMM framework. Overall, the paper is well written, and it is theoretically convincing. The experimental study could be more solid, e.g., it is reasonable to have a word-level baseline, as the proposed approach lies in between the character-level and word-level systems. the vocabulary size of the WSJ si284 dataset is 20K at maximum, which is not very large for the softmax layer, and it is a closed vocabulary task. I guess the word-level system may be also competitive to the numbers reported in this paper. Furthermore, can you explain what is the computational bottleneck of the proposed approach? You downsampled the data by the factor of 4 using an RNN, and it still took around 5 days to converge. To me, it is a bit expensive, especially given that you only take one sample when computing the gradient. Table 2 is a little bit misleading, as CTC with language model and seq2seq with a language model model from Bahdanau et al. is much closer to the best number reported in this Table 2, while you may only get a very small improvement using a language model. Finally, *O(5) days to converge* sounds a bit odd to me.
assistant
This submission proposes to learn the word decomposition, or word to sub-word sequence mapping jointly with the attention based sequence-to-sequence model. A particular feature of this approach is that the decomposition is not static, instead, it also conditions on the acoustic input, and the mapping is probabilistic, i.e., one word may map to multiple sub-word sequences. The authors argue that the dynamic decomposition approach can more naturally reflect the acoustic pattern. Interestingly, the motivation behind this approach is analogous to learning the pronunciation mixture model for HMM based speech recognition, where the probabilistic mapping from a word to its pronunciations also conditions on the acoustic input, e.g., I. McGraw, I. Badr, and J. Glass, *Learning lexicons form speech using a pronunciation mixture model,* in IEEE Transactions on Audio, Speech, and Language Processing, 2013 L. Lu, A. Ghoshal, S. Renals, *Acoustic data-driven pronunciation lexicon for large vocabulary speech recognition*, in Proc. ASRU R. Singh, B. Raj, and R. Stern, *Automatic generation of subword units for speech recognition systems,* in IEEE Transactions on Speech and Audio Processing, 2002 It would be interesting to put this work in the context by linking it to some previous works in the HMM framework. Overall, the paper is well written, and it is theoretically convincing. The experimental study could be more solid, e.g., it is reasonable to have a word-level baseline, as the proposed approach lies in between the character-level and word-level systems. the vocabulary size of the WSJ si284 dataset is 20K at maximum, which is not very large for the softmax layer, and it is a closed vocabulary task. I guess the word-level system may be also competitive to the numbers reported in this paper. Furthermore, can you explain what is the computational bottleneck of the proposed approach? You downsampled the data by the factor of 4 using an RNN, and it still took around 5 days to converge. To me, it is a bit expensive, especially given that you only take one sample when computing the gradient. Table 2 is a little bit misleading, as CTC with language model and seq2seq with a language model model from Bahdanau et al. is much closer to the best number reported in this Table 2, while you may only get a very small improvement using a language model. Finally, *O(5) days to converge* sounds a bit odd to me.