An Online Sequence-to-Sequence Model Using Partial Conditioning
Sequence-to-sequence models have achieved impressive results on various tasks. However, they are unsuitable for tasks that require incremental predictions to be made as more data arrives or tasks that have long input sequences and output sequences. This is because they generate an output sequence conditioned on an entire input sequence. In this paper, we present a Neural Transducer that can make incremental predictions as more input arrives, without redoing the entire computation. Unlike sequence-to-sequence models, the Neural Transducer computes the next-step distribution conditioned on the partially observed input sequence and the partially generated sequence. At each time step, the transducer can decide to emit zero to many output symbols. The data can be processed using an encoder and presented as input to the transducer. The discrete decision to emit a symbol at every time step makes it difficult to learn with conventional backpropagation. It is however possible to train the transducer by using a dynamic programming algorithm to generate target discrete decisions. Our experiments show that the Neural Transducer works well in settings where it is required to produce output predictions as data come in. We also find that the Neural Transducer performs well for long sequences even when attention mechanisms are not used.
The authors introduced an encoder-decoder architecture with attention over blocks of input and variable length decoder structure. The encoder is a multi-layer LSTM RNN. The decoder is an RNN model conditioned on weighted sums of the last layer of the encoder and it's previous output. The weighting schemes (attention) varies and can be conditioned on the hidden states or also previous attention vectors. The decoder model produces a sequence of symbols, until it outputs a special end character "e" and is moved to the next block (other mechanisms where explored as well (no end-of-block-symbol and separately predicting the end of a block given the attention vector). It is then fed the weighted sum of the next block of encoder states. The resulting sequence of symbols determines an alignment of the target symbols over the blocks of inputs, where each block may be assigned a variable number of characters. The system is trained by fixing an alignment, that approximately resembles the best alignment. Finding this approximately best alignment is akin to a beam-search with a beam size of M (line 169), but a restricted set of symbols conditional on the last symbol in a particular hypothesis (since the target sequence is known). Alignments are computed less frequently than model updates (typically every 100 to 300 sequences). For inference, an unconstrained beam-search procedure is performed with a threshold on sequence length and beam size.
First of all, I think the paper is well written and clear.  Judging by the title the main contribution of this model is a novel model architecture. Looking at the references, there have previously been speech recognition systems, that use attention ([2] and [5]). Sequence transduction has also been introduced by Graves in "Sequence Transduction with Recurrent Neural Networks" in 2012. The follow up paper is cited as [7] uses this transduction method. Grave's describes his transducer to "extend CTC by defining a distribution over output sequences of all lengths, and by jointly modeling both input-output and output-output dependencies.". To me the novelty of this paper then is the idea of applying the transducer on a block-by-block basis and defining a RNN conditional on it's last definite output. Other methods integrate over alignments using the output of an RNN, whereas here the authors define a separate complex network structure to perform the transduction. However, Graves and Hinton also use a separate prediction network to feed into the transduction graph. To me, the disadvantage of the method presented here is the complicated training procedure, since exact constrained inference becomes intractable. I'm am also not sure how to judge the novelty of this approach, because I might be missing literature, since there has been a lot of developments in this area. It seems very similar to Graves and Hinton [7], especially since they also use the separate prediction network, except for the attention mechanism (but in later experiments we see, that the attention mechanism doesn't necessarily help that much (unless larger window sizes have advantages (one of which might be speed))).  The experimental evaluation of this method is performed on an addition toy task and TIMIT. For the addition task the authors constrain themselves to adding two three-digit numbers. It would be interesting to see how the model generalizes to longer numbers. That is, if trained on additions of three-digit numbers, how well does it add four-digit numbers?  For TIMIT the authors first compare having recurrent state across blocks, vs. not having a recurrent state. I think, if you don't have recurrent state in the transducer, you might be able to perform more efficient inference using Graves' transduction method. I'm not too sure on this point. Figure 3 shows that for some block sizes no attention performs equally well. I think no-attention means, that the encoder hidden states are averaged per block and then fed to the transducer. This is then akin to a convolution of size W and stride W. Maybe it is interesting to see how the encoder would perform, if W states are combined and fed into a simple RNN, that produces a softmax over symbols, which is then trained with CTC or Graves' Transducer. The best performance of Graves' Transducer on a unidirectional LSTM RNN has been reported at 19.6%, which is close to the best 19.8% reported here, albeit I understand, that the authors argue, that this model needs to be tuned more, which can be in itself a lot of work. Their best network uses a 3 layer LSTM RNN transducer. I think four layer transducers might vary much in accuracy, because they start overfitting a lot on TIMIT. It would be interesting to see how much these models overfit.   Overall, given the complexity of the model and the performance, plus similarities to other methods, I'm not very sure how interesting this method is (I might lack the breath in literature to point out exactly how similar this model is). I would be interested in understanding how exactly it differs from Graves' and Hinton's Transducer network. I am not very sure on my review of the novelty of this method. The performance in itself is competitive on TIMIT, but other, simpler methods perform equally well. I am not sure if the extra capacity and context is necessary in comparison to other methods or for this dataset. Perhaps, if the model was trained on more data, it would become more apparent, but I'm not sure how well it scales. I wonder how long it takes to fully train a model?